Methodological Note: On Using Personal Digital Assistants (PDAs) for Survey Administration in the Study of Youth Development

Applied developmental scientists face the challenge of identifying research methods that enable the efficient collection of data from youth of diverse social backgrounds (e.g., ethnic. racial, religious, economic) and varying levels cognitive-linguistic and attentional skills. In addition, because access to youth during school time is often limited by educators’ desire to preserve instructional time, finding methodologies to collect data from youth that are highly efficient, and also those that are feasible in less structured settings, are needed. This article outlines some of the benefits and limitations of using a voiceenhanced survey delivered on a personal digital assistants (PDA) as a method of gathering data from diverse youth in both, in and out-ofschool contexts.


Introduction
Applied developmental scientists face the challenge of identifying research methods that are both motivating and efficient with respect to the collection of data. Such methods need to be identified for use in schools and out-of-school contexts from youth of diverse backgrounds (e.g., ethnic. racial, religious) and varying cognitive-linguistic and attentional levels. Access to youth during school time is often limited by school systems' desire to maintain the integrity of instructional time, particularly given the high-stakes testing and performance-related assessments of students, teachers, and schools. Access to youth during out-of-school time may be limited by youth's engagement in a range of structured and unstructured activities. A typical paper-and-pencil (PaP) version of a survey that contains approximately 200 questions, for instance, can take between one and 2.5 hours to administer if students are read the survey aloud or if they are asked to read through the survey themselves.
Regardless of the in or out-of-school context involved in a given study, researchers are constrained by the content, pace, and format of the survey. The lack of control that youth have over the pacing of surveys that are read aloud to them, and the cognitive and attentional demands that reading the surveys themselves entails, make both of these methods of survey administration non-optimal, especially among participants for whom English is a second language (Trapl, Borawski, Stork , Lovegreen, Colabianchi, Cole & Charvat, 2005). Time may be a limiting factor, especially if working within the 50 minute structure of secondary school classes. In addition, many "surveys" seem like "tests" because of the way they are formatted (e.g., "fill in the bubbles"). Finally, to the extent surveys assess personal topics related to mental health, health and behavioral choices, privacy concerns are important determinants of participant responses. In sum, due to a range of cognitive-attentional, motivational, and privacy-related issues, students often skip questions or fail to respond in ways that accurately reflect their views and experiences on traditional surveys.
A voice-enhanced personal digital assistant (PDA) assessment may provide a useful alternative to the PaP method in accessing youth during school and out-of-school time. This article outlines some of the benefits and limitations of the PDA as a method for gathering data with diverse youth.

Voice-enhanced, PDA-delivered surveys
One method that has been proposed to address these issues associated with PaP surveys is the use of voice-enhanced surveys delivered on a personal digital assistant or PDA. While there is a range of PDA and PDA-like products available, PDAs share several common characteristics: they are (a) very powerful computers running the Windows operating system that have the potential for enhanced memory capability through memory cards; (b) easily held in one hand; (c) capable of running on battery power for relatively longer periods of time than most laptop computers; (d) utilize a pen or stylus; and (e) offer a range of communication and software capabilities (Bayus, Jain & Rao, 1997).
PDAs have been used in Africa for research on health and in the United States for research on risk behavior (see Center for Disease Control and Prevention, 2005;Trapl et al., 2005). The technical benefits of PDAs include streamlined data entry and data cleaning processes, as well as fewer problems with missing data generally or resulting from complicated skip patterns.
To create surveys to be delivered on the PDA platform with its small screen size, specialized software is needed. We used Dell Axim X51 PDAs and a program called "SEDCA" Soundenhanced Data Collection Application (http://www.dontpapanic.com/). The textual version of the survey is designed using this specialized software. In addition, it is necessary to record mp3 digital voice files for each of the survey questions (and/or responses). We hired a student actress, obtained a soundproof room in the Tufts Radio station, and used basic microphone and recording software to record the voice files for each survey question directly onto a laptop computer in mp3 format.
These audio files are then integrated with the actual written survey items through the specialized software program. The final survey, as well as all of the individual voice files, are then saved on extra memory cards that are inserted into each PDA. It is possible to have various language versions of the written surveys as well as the audio tracks of the survey on the PDA at one time using this methodology. However, memory limitations and the size of the audio files constrain this possibility.
After the design process and loading of the survey is completed, each PDA, with a pair of headphones, is now ready to deliver the voice-enhanced survey. Participants use a wand and a set of arrows to navigate through the survey on the PDA screen. Each time participants click to a new question, that question appears and is read aloud to them. Participants can adjust the volume and they have the ability to hear questions again. Participants click on their responses on the screen with the wand and they move through the survey at their own pace, in privacy.

Method
To illustrate our own experience of the benefits of using voice-enhanced, PDA-surveys, we describe data that we gathered as part of a larger study sponsored by the John Templeton Foundation (JTF) entitled "The Role of Spiritual Development in Growth of Purpose, Generosity, and Psychological Health in Adolescence." This project was a cross-sectional and multi-method study conducted in the greater Boston area (Lerner, Roeser, & Phelps, 2008). In a small methodological case study that we undertook as part of this larger study, we compared participants' experiences in taking a pencil-and-paper version of the study survey with their subsequent experience in taking the same survey using the voice-enhanced version delivered via PDA. Following the second administration of the survey in the PDA format, we conducted short exit interviews with participants who took both versions in order to elicit their views about the two methods. We present below some quotes from these interviews, as well as our own observations regarding the use of the two methods.

Sample
The sample was derived from a cross-sectional study of 399 participants (55.7% male) between the ages of 10 to 23 years who were from private religious schools and youth groups, public schools, and community-based programs in the Greater Boston area. Data were drawn from the above-noted JTF study; 48.0% of the participants were in middle school, 47.1% in high school, 2.8% in college or of traditional college age, and 2.1% either below or above the age range. The participants were religiously and ethnically diverse. In addition, 67% of participants reported that they belonged to immigrant-origin families.
The case sample included a PaP group of 31 participants (14 males, 17 females) with an age range from 11 to 21 years. The PDA sample involved a group of 24 participants (15 males, 9 females), with an age range of 12 to 26 years. The qualitative data presented here come from exit interviews we conducted with 10 participants who completed the PaP and who returned three months later to complete the PDA version of the same survey (6 males, 4 females).

Procedure
Administration of the paper-and-pencil (PaP) version of the survey was completed in December 2006. Youth participants were given an overview of the survey and assistance by the researchers in completing sample questions in order to familiarize them with the item response formats (e.g., multiple choice, true/false, Likert-scale items). They then were asked to complete the survey on their own, asking questions of the research assistants as needed. The second administration was completed nearly three months later. Participants were given a PDA, a pair of head phones, a stylus and instructions about how to complete the survey at their own pace.
Once they learned how to use the PDA, they set to work on the survey. Incentives for survey completion were offered at both times, and included pizza, the headphones used with the PDA, and the raffling off of an I-Pod Nano.

Results
Observational data from research assistants and timestamps on the PDA-surveys showed that the administration time for the PaP and PDA-versions of the survey varied considerably. In general, we found that the PDA-version of the survey took about 30% less time. Furthermore, as evidenced by the noise level and frequency of interruptions that occurred during the longer time it took to complete the PaP version, and the relative silence and shorter time it took for participants to complete the PDA version, it seems reasonable to infer greater motivation, attention, and on-task behavior in the PDA vs. the PaP version. We also observed that there was significant confusion regarding the flow of prompts and skip-patterns with the PaP version for several items on the survey. However, complicated skip patterns were more easily navigated with the PDA because of the automation of such patterns. It could well be that testretest effects shaped greater on-task behavior and shorter times to complete the survey on the PDA, but we believe that (a) the motivating and private nature of the method and (b) the reduction in cognitive/attentional load may have also contributed these differences.
To assess these conjectures, we asked the 10 individuals who had taken both version of the survey which method they found easier to navigate. We found that all 10 participants who responded to both formats indicated that they considered the PDA-version easier. Although the PaP and PDA-versions of the surveys contained the same content and number of items, participants perceived the PDA administered survey as easier, "Because you see all those papers and you'd be like, 'Aah. It's so much.' And you'd see that it's [the PDA] and you think, that will be faster." When asked which survey method they preferred and would recommend for use with other students, one participant noted that, "It (the PDA) was much easier instead of writing because it was clear. It's better for younger kids because they can understand better most of the time." In comparison with the PaP version, one participated preferred the PDA; "Because it was better. I didn't feel overwhelmed with all the papers. So it was better." The perceived excessive length of the PaP version may have affected the amount of missing data, as evidenced by the observation that, "…on the last test I skipped a lot of questions because I didn't feel like reading it." It is interesting to note her calling the PaP version of the survey a "test." Another participant stated that the PDA was "better, you didn't have to write." When asked whether the method of administering the survey affected participants' answers, participants had mixed perspectives; this feedback points to some limitations of the PDA. Some participants suggested that despite the increased cognitive load, the PaP version may promote more reflective participation, "Because you have to read it, and you have to take your time….and the PDA, you could just rush through and put any answer down." Another participant provided an alternative interpretation that, "You could rush through the one on the paper too." The participant who considered the PaP version as fostering more thoughtful participation concluded that, "Yeah, but it would be more intimate to read it." One participant suggested that because there is less cognitive effort involved in reading and writing, a participant's full cognitive capacity may be engaged with the PDA, "The PDA is better because it tells you, you have more [time] to think clearly." That is, this participant believed that working memory capacity was freed up in the PDA-version, and thus this capacity was available to reflect on the question and answer it more "clearly."

Discussion
Consistent with other research on the use of PDA-versions of surveys with adolescent populations (Trapl, et. al, 2005) we found that the PDAs were very well-received by our participants. Survey completion time was decreased by approximately 30%. Reduced completion time for the PDA-version also meant that the survey was less of an imposition on non-instructional time of participants. The structured format of the PDA also allowed easier negotiation of questions that involved complicated response patterns. The voice-enhancement also seemed to reduce cognitive load.
Based on exit interviews, we found that the participants unanimously considered the PDA as easier, faster, and more accessible. While most participants recommended the PDAs for future administration of the survey, there were mixed opinions about which method would provide responses that best reflect what students believe. Some participants maintained that PDAs provided an easier engagement strategy making it accessible, but at least two participants suggested that people could just "tap" too fast, and therefore not really think about their answers.
Of course, this examination of the benefits and limitations of PDAs is based on a case study at a particular research site. The small sample size precluded a full analysis of missing data, patterns of variance within the aggregate sample, and whether the general aversion to long PaP versions of survey would be equally shared with participants who have higher English proficiency or higher proficiency in reading and writing, this was the case with our sample.
In turn, while the PDA does afford numerous technical benefits, such as relatively easy downloading of data with limited data entry or cleaning, PDAs do require financial and temporal investments, as well as maintenance. There are significant start-up costs associated with purchasing and programming a sufficient number of PDAs to support simultaneous data collection in multiple classrooms. In addition, the use of PDAs also requires that plans be made for their recharging, security, transportation, and technical support so that they may be reused in a research setting even if the particular PDA model is no longer available for purchase on the market.
In sum, PDAs are a viable alternative to paper and pencil versions of surveys for participants in a range of in-school and out-of-school settings, and should be investigated by others for use in youth development research. Voice-enhanced PDAs reduce the cognitive load for participants, increasing the engagement of the participant in the survey completion, and reduce survey completion time. Furthermore, researchers are beginning to go beyond examining how PDAs reduce missing data to evaluate the quality of the data (e.g., reliability) (e.g., Trapl et al., 2005). We believe PDAs will remain an important alternative to PaP versions of surveys, but more methodological research on the issues raised in this report is needed. Future research may consider documenting whether there are significant reductions in missing data with participants from a range of demographic groups, so as to best specify with which populations the PDA may grant the highest margin of benefits.