Resource Review—Using Qualtrics Core XM for Surveying Youth

Qualtrics software, a digital tool used for survey data collection and analysis, has become a commonly used technology in organizations around the country. Many educators are using this tool to collect, analyze, and communicate education, outreach, and engagement evaluation data in an efficient and effective way. In a youth development context, Qualtrics Core XM software can be used to design youthfocused digital surveys that are engaging and efficient, and it can help build program evaluation capacity. An assessment of some key functions of this tool and their applications to youth development program contexts is presented here.


Introduction and Background
In recent years, Qualtrics software has become ubiquitous (Rogers, 2017) and youth development professionals around the country are using this tool to conduct program Journal of Youth Development | http://jyd.pitt.edu/ | Vol. 16 Issue 1 DOI 10. 5195/jyd.2021.886 Using Qualtrics Core XM for Surveying Youth 162 evaluation (Walahoski, 2014). Over the past 3 years we have used the Qualtrics Core XM tool to (a) create customized evaluation instruments, (b) improve data collection efficiencies, and (c) increase quantitative data analysis capacity. In assessing our use of Qualtrics Core XM software for these purposes, we identified a series of considerations that may be of interest to other youth development professionals. It should be noted that Qualtrics software has a myriad of features, and a number of products are available. In the discussion that follows we will highlight some of the key features we have used to support our youth development evaluation goals.

Using Qualtrics to Create Customized Evaluation Instruments
There are a variety of features within Qualtrics Core XM that allow for survey instruments to be customized for youth audiences. For example, using skip and display logic functions, surveys can be customized and tailored to particular audiences or age groups, or to a variety of other characteristics or attributes (e.g., program participation levels, hours of intervention exposure) ( Figure 1). Doing so allows for a more streamlined survey-taking experience for the participant (Hill, 2013). For example, we have used this feature to restrict college and career readiness questions to youth in Grades 9 through 12.

Figure 1. Example of Skip and Display Logic Function
Additionally, Qualtrics Core XM allows for customized end-of-survey messages ( Figure 2). This feature can be used to direct participants to a website or other web resources of interest or to send customized thank you messages to youth or their parents/guardians (Qualtrics, 2019; Journal of Youth Development | http://jyd.pitt.edu/ | Vol. 16 Issue 1 DOI 10.5195/jyd.2021.886 Using Qualtrics Core XM for Surveying Youth 163 Figure 3). Moreover, having these customized messages helps with providing participants with individualized reasons regarding why they were filtered out of the survey, thus avoiding calls and emails from participants who may believe they were inaccurately screened (e.g., because they did not participate in at least 5 hours of programming).

Figure 3. Example of Customized End-of-Survey Message
The contact list feature is an essential element of the system due to its usefulness in improving the efficiency of data collection. By importing a .csv file of participants' names from an enrollment management system as a contact list, individualized survey links can be quickly emailed to all or some of the parents/guardians or youth. Furthermore, these messages can use the "piped text" feature, which allows for a youth's or parent's/guardian's name to be embedded in the greeting line ( Figure 4). We found this feature especially helpful, as some families have more than one youth using the same email address, and this allows us to differentiate which link is intended for each youth in the family. Furthermore, this feature helps with improving response rates and getting broader participation in evaluation data collection efforts (Tourangeau & Plewes, 2013  When using the contact list feature, data managers can easily oversee who was contacted, when they were contacted, and if they opened the URL or not. Similarly, by using the contact list feature, the Qualtrics Core XM system can automatically generate reminder emails to only those individuals who have not yet completed the survey. To increase response rates, reminders can be personalized and set with a customized frequency or set for a specific date (Monroe & Adams, 2012).
Another important feature for youth development professionals is the use of contact lists for longitudinal tracking. This can improve efficiency, especially in programs designed for long-term engagement of youth across multi-year enrollments. Long-term tracking of youth progress is important for demonstrating the impact of a program or set of programs on youth over time. To this end, we have initiated an annual evaluation program that uses this feature to measure changes in youth responses over multiple years.

Using Qualtrics to Increase Data Analysis Capacity
The statistical analysis tools in Qualtrics Core XM can be used to simply analyze and visualize survey data ( Figure 5). This technology supports increased evaluation capacity by allowing youth development professionals to analyze basic quantitative data without the need for advanced statistical analysis training (e.g., SPSS or SAS statistical packages). However, although the basic analysis features are easier to use than more advanced programs, training individuals in how to use the tools effectively is still necessary. Additionally, for more rigorous analyses, someone with advanced statistical assessment knowledge will likely be needed to help in the data analysis and synthesis processes, especially if academic publishing is a goal.

Considerations for Youth Development Professionals
The Qualtrics system can be used to support broad and inclusive participation in evaluation efforts within youth development organizations. Using the Qualtrics Core XM software, youth development professionals can • Design customized survey experiences, improve the efficiency of data collection processes, and increase their capacity to measure short-and long-term impacts of programs and intervention efforts.
• Increase their options for survey design with careful consideration of survey design standards and integration of the functional features described above.
• Efficiently notify participants of surveys, remind them of surveys in progress and track survey completion. • Streamline program evaluation and data analysis across program types and/or conduct multiple evaluations at once.
• Generate data visualizations from Qualtrics Core XM surveys. For example, they can be used to communicate results to stakeholders and train volunteers and community partners (O'Connor & Zeldin, 2005).
What follows are a few recommendations to consider when using Qualtrics Core XM software.
• In our experience, some emails may be blocked by the Qualtrics server; therefore, monitoring the distribution of survey emails will allow confirmation of email delivery.
• In order to streamline the approval process, individuals may want to design their survey prior to obtaining any necessary Institutional Review Board approval.
• Youth development professionals should follow guidelines for survey design and implementation as recommended by experts such as Dillman et al. (2014), Fowler (2009, and Rea and Parker (2005).
• Keep your organization's accessibility guidelines in mind when creating digital surveys or visualizations in order to maximize accessibility.

How to Obtain
Check with your organization or institution to find out if a site license is already available to you.
If you do not already have access, a 30-day free trial version of Qualtrics is available in addition to free consultative demonstrations of the tool. Qualtrics can provide individualized pricing for the various products offered based on users' needs. Qualtrics also provides training online that can be found at www.basecamp.qualtrics.com. They also have a direct contact number (1-800-340-9194) with customer service support and survey assistance. More information can be found at www.qualtrics.com.