Including the Youth Perspective : The Development of the CPQA Camper Survey

The purpose of this study was to pilot the use of the CPQA Camper Survey, a camper self-report survey to assess summer camp program quality. The survey is based on the best practices identified in the American Camp Association’s Camp Program Quality Assessment (CPQA) short form (American Camp Association, n.d.). Best practices are organized into 5 subscales on the CPQA: staff behavior; emotional safety; camper choice, planning, and reflection; learning at camp; and nature. The CPQA Camper Survey asked youth campers at 5 different overnight camps to report on their perceptions of how often camp program quality best practices occur at summer camp. Results from the surveys collected showed that for each of the 5 subscales, campers’ average camp program quality ratings tended to cluster around the upper end of the rating scales. Respondents’ answers were consistent across the questions comprising each subscale, which indicates that the questions in each subscale reliably measure the same construct or idea. The results of this study show that the CPQA Camper Survey is one tool that camp directors and administrators can use as part of their program improvement processes to assess the quality of their programs, and thus improve the quality of the camp experience.


Introduction
The purpose of this study was to pilot test the use of a camper self-report survey to assess summer camp program quality.This study asked youth campers to report on their perceptions of how often camp program quality best practices occur at summer camp.The results of this study can help camp programs to improve the quality of the camp experience by providing a youth self-report tool that can be used by camps in program improvement processes.
Camp experiences have long been thought to lead to positive youth development outcomes in young people (Bialeschki, Fine & Bennett, 2016).Camp programs have been recognized as helping youth to increase confidence and self-esteem, develop social skills, make new friends, grow more independent, develop leadership qualities, and become more adventurous and willing to try new things (American Camp Association, 2005).Youth development research has shown the importance of positive youth development outcomes, and has identified strategies youth programs can utilize to improve programming in order to increase outcome achievement (Durlak, Weissberg, & Pachan, 2010;Lerner et al., 2005;Peisner-Feinberg et al., 2000;Reisner, White, Russell, & Birmingham, 2004;Shernoff & Vandell, 2007).Research also indicates that the design and implementation of youth programs, as well as the characteristics of individual youth, can have an effect on the achievement of desired outcomes (Bronfenbrenner & Morris, 2006;Eccles & Gootman, 2002;Lerner et al., 2005;Pittman, Irby, Tolman, Yohalem, & Ferber, 2003).While the characteristics of individual youth are often outside the control of program directors and staff, areas of program design and implementation are within their sphere of influence.
Camp programs for youth can be intentionally designed and implemented in ways that have been shown to be effective.This intentional design often includes practices that are typically employed by programs attaining high levels of outcome achievement.The ability of youth programs to implement recognized best practices in their field is often expressed in terms of program quality.High quality programs include elements such as the performance and behavior of program staff; the existence of positive and supporting relationships; youth feeling a sense of belonging and developing self-efficacy; the availability of opportunities for active learning and the acquisition of new skills; and youth having the ability to make decisions, develop a sense of independence, and have a voice (Bowles & Brand, 2009;Durlak, et al., 2010;Grossman, Campbell, & Raley, 2007;Larson, Eccles, & Gootman, 2004;Larson, Rickman, Gibbons, & Walker, 2009;Sibthorp, Paisley & Gookin, 2007;Smith et al., 2009;Vandell, et al., 2007).In general, higher quality programs are those that are believed to produce increased outcome achievement, thus having a positive impact on the lives of youth (Garst, Browne & Bialeschki, 2011;Sheldon, Arbreton, Hopkins, & Grossman, 2010;Smith, Devaney, Akiva, & Sugar, 2009).
High quality programs accomplish this by enhancing the potential that youth participants will be sufficiently engaged in order to take advantage of any educational and developmental opportunities that are presented (Vandell, Reisner, & Pierce, 2007).
Many camps work to create high quality program environments that help them reach their intended outcomes.Typically, the quality of a program environment is assessed through program observation.Unfortunately, observational assessments can require a large financial investment and be difficult to implement (Rowan, Jacob, & Correnti, 2009).In order to mitigate these issues, there is a need for alternative methods of assessing youth program environments (Morgan, Sibthorp, & Browne, 2016).Considering the importance of including the perspectives of youth participants in program decision-making (Akiva, Cortina, & Smith, 2014), there exists an opportunity for camp programs to include alternatives to observational assessments that include the perspective of youth.
In order to include the youth perspective in observational assessments of program quality, this study sought to develop a self-report instrument that can be completed by youth attending summer camp programs.In partnership with the American Camp Association's Not-For-Profit Council, the researchers in this study developed and pilot tested the CPQA Camper Survey during the summer of 2016.Survey responses were analyzed for reliability, validity, and ease of use.Respondents were asked to rate, on a scale of 0 to10, with 0 being "this never happens" and 10 being "this always happens," how often they perceived that each best practice occurred at camp.Minor word changes were made to the CPQA Best Practices to address the perspective of youth attending camp.For example, campers in this survey were asked to rate how often "I have opportunities to . . .," whereas the CPQA Best Practice begins with "Campers have opportunities to . . .."

Participant Selection
Invitations to participate in the project were sent to 25 camps that had previously participated in an American Camp Association research study.Each camp was asked to identify 50 campers between the ages of 10 and16 years to complete a camper survey.An online project information meeting was held in May 2016 to explain the project to interested camps and answer any questions people may have.

Survey Administration
Camps were instructed that the CPQA Camper Survey should be administered toward the end of a camp experience, preferably the second-to-last day.Additional guidelines given to camps for administering the survey are included in the survey administration preparation section of the survey (see Appendix A) and include:  Instructions should be read aloud by an adult staff member who is familiar with the survey.
 Campers should be seated at a comfortable table (or tables) with space for them to write numerical scores on a piece of paper.
 Each camper should be given a CPQA Camper Survey packet and a pencil.Since many of the questions are about camp staff, camps were asked to make sure that the counselors were not able to see the camper surveys.Camps were encouraged to make arrangements ahead of time to create an atmosphere in which campers felt safe to answer questions honestly.
Closely adhering to the administration instructions can help to mitigate some of the limitations of self-report surveys.Self-report tools are known to be susceptible to different sources of error, but the majority of the errors can be attributed to the nature of the instrument and the individual's subjectivity when interpreting and completing the questionnaire (Saint-Maurice & Welk, 2014).In addition to ensuring campers felt safe in answering the survey questions, the nature of the instrument was addressed by wording the questions appropriately and making them easy to understand.

Data Validation
The items for the CPQA Camper Survey were assessed for overall normality of the distributions and potential ceiling or floor effects.Each subscale was assessed for internal consistency reliability using Cronbach's alpha.The five subscales were also examined for correlations within campers.Finally, an exploratory factor analysis was conducted to investigate the underlying constructs related to the content validity of the instrument.A pre-determined eigenvalue (λ > 1.0) and suppression level (0.3) was utilized to allow the items of the survey to reduce into the number of factors meeting these criteria.
A total CPQA Camper Survey rating score was calculated for each subscale by adding up the total score for each subscale and dividing by the number of questions in the subscale.For this study, cases were excluded listwise, meaning that if a survey had missing data on a variable being assessed, it was not included in the analysis of that particular subscale..

Distribution of the Data
Overall, data from the CPQA Camper Surveys were negatively skewed, meaning that the majority of the total rating scores for each subscale tended to be grouped around the upper portion of the rating scale.Subscale means ranged from 7.97 for choice, planning, and reflection to 8.76 for emotional safety.Descriptive statistics for each subscale are summarized in Table 1.This can make it difficult to assess the unique influence of different areas of program quality.It also suggests that if camps can "move the needle" in one area, then the perception that campers have of the overall quality of the camp experience could increase as well.Correlations of survey subscales are presented in Table 3.

Exploratory Factor Analysis
Exploratory factor analysis was used to assess the underlying constructs related to the CPQA Camper Survey.Tests for factorability of the data resulted in a high KMO score (.911) and Bartlett's test of sphericity was significant (p < .01).These results indicate that the scores of the survey are considered suitable for factor analysis.
Principal-axis factor analyses with direct oblimin rotation was conducted.The number of factors was determined by setting eigenvalues greater than 1.0 and a cut-off suppression value of 0.3.
Exploratory factor analysis yielded seven factors.Factor 1 explained 38.5% of the variance.
Factors 2 through 7 combined explain an additional 20.4% of the variance.These results mean that most of the explained variance in responses amongst campers can be attributed to Factor 1. Percentages of the explained variance attributed to each factor are presented in Table 4.
Analysis of the factor loading scores attributed to Factor 1 provide insight into the validity of survey instrument.In general, factor loadings above .6are considered the most representative.
Factor 1 loading scores indicate that the most representative items are the fifth, sixth, and seventh question in the Learning at Camp subscale.Looking back at these subscale items reveals that the majority of the variance in camper scores can be attributed to the best practices of staff asking challenging questions, youth having the experience of collaborating with others, and youth having opportunities for their brains to be active.Factor loading scores can be found in Appendix B. Each of the CPQA Camper Survey subscales was found to be reliable, meaning that the answers of respondents were consistent across the questions comprising each subscale.This indicates that camps can feel confident that the survey subscale questions reliably work together to measure each construct or idea.
The subscales also were significantly correlated with each other, meaning that campers tended to score all of the CPQA Camper Survey subscales similar to each other.These correlations can make it difficult to assess the unique influence of each particular subscale area, but also indicate that if camps can improve quality in one area, overall perceptions of camp program quality may also increase.
For the camps in this study, exploratory factor analysis of the CPQA Camper Survey scores indicated that 38.5% of the variance in camper scores can be attributed to Factor 1. Factor 1 is most represented by the best practices of (a) staff asking challenging questions, (b) youth having the experience of collaborating with others, and (c) youth having opportunities for their brains to be active.These results indicate that an opportunity exists for camps to improve the frequency these particular best practices occur at camp.
Overall, the factors revealed through exploratory factor analysis do not necessarily align with the CPQA program quality areas.Refining the subscales to more closely match the factor loading scores could improve the validity of the survey instrument.Validity is something that becomes clearer over time and through continued use, so it is recommended that future research utilizing the CPQA Camper Survey also includes assessments of validity.
Exploratory factor analysis also indicated that questions having to do with individual camper experiences loaded together, while questions dealing with camp structure and organization loaded together.Many of these items are grouped together in the current CPQA domains.
Splitting them apart into separate subscales could improve the performance of these instruments, but would be different from the organization of the CPQA Short Form and full CPQA observation instrument, thus limiting opportunities for comparisons.For this reason it is recommended that the current subscale structure be retained.
In conclusion, camps seeking to include the perspective of youth in camp program quality assessment and improvement processes are encouraged to utilize the new CPQA Camper Vandell, D., Reisner, E., & Pierce, K. (2007).Outcomes linked to high-quality afterschool programs: longitudinal findings from thestudy of promising afterschool programs.Irvine, CA, Madison, WI: Madison and Policy Studies Associates.

PHASE 3: Campers Read and Answer Questions
The Survey Administrator can answer questions from individual campers throughout the survey.It is ok to read questions to campers and to help explain any terms that they may not understand.
Please refrain from suggesting any type of answers for campers.For example, avoid saying things such as "I think that should be this number" or "how about this choice?"Please refrain from questioning or commenting on any of the answers selected by campers.We want the answers to be freely chosen by the campers themselves.
We anticipate that the survey will take most campers around 15-20 minutes to complete.Please don't rush campers that go slower or try to make them stay until they are done if they don't want to.The format of the survey can be a little difficult for some campers, so please do what you can to help them out while not providing answers.

PHASE 4: Collecting and Analyzing Data
Collect the answer sheets from all campers.Invite them to ask questions.Answer their questions as completely as possible, and then thank campers for helping to make camp a better experience for future campers by contributing their thoughts.
To analyze the data, begin by inputting survey scores into a spreadsheet.For each camper, calculate the average score for each section.Add the scores in each section together, and then divide this total by the number of questions in the section.
Calculate the overall average score for your camp in each section.Use these scores as part of a program improvement and/or staff-training program.Rather than focusing on whether or not the campers are correct in their assessments, consider these scores as baselines.If there is an area you want to improve, than implement changes and survey campers again the following year.Compare the results.
The Camper Perceptions of Program Quality Survey is best utilized as part of an overall program quality assessment and program improvement.The purpose of this survey is to help camp administrators to include the perspective of youth in their assessment process.This survey is best used as a supplement to a formal program quality observation and other forms of CPQA self-assessment.
For more information on camp program quality assessment, visit the American Camp Association website (www.acacamps.org)and search on "CPQA."

American Camp Association CPQA Camper Survey
This survey will ask you about your experiences while at camp.The format of this survey is a little different -so, please take a moment to look at the sample questions below.

Instructions
For each question, the survey will ask you to please estimate, on a scale of 0-10, with "10" being "this always happens" and "0" being "this never happens", how frequently the following things occur at camp.

A Couple of Examples
Please write the number for your response in the box next to each statement: Staff use a warm tone of voice and respectful language.8  An answer of "8" indicates that you think staff use a warm tone of voice and respectful language most of the time.
I am encouraged to try out new skills or attempt higher levels of performance. 6  An answer of "6" indicates that you feel like you are encouraged to try out new skills or attempt higher levels of performance a little more than half of the time. As

Section I: Staff Behaviors
Please estimate on a scale of 0-10, with "10" being "this always happens" and "0" being "this never happens", how frequently the following occur: 0 to 10 Staff use a warm tone of voice and respectful language.
Staff smile, use friendly gestures, and make eye contact.
When campers approach them, staff are attentive and responsive.
Staff circulate (and spread out if multiple staff) to interact with every camper (in groups or individually) at some point during every activity.
Staff interact individually at least once with every (or almost every) camper during every activity.
Staff are actively involved with campers.(e.g., they provide directions, answer questions, work as partners or team members, check in with campers).

Section II: Emotional Safety
The next set of questions relates to emotional safety and support for belonging.How frequently would you say the following occur: 0 to 10 Staff show respect for all campers and insists that campers show respect for each other.(e.g., use kind words, take turns, help each other) Staff address any incidents in which a camper or campers are made fun of.
I feel free to be myself.
When there is a conflict or an incident involving strong feelings, staff ask about and/or acknowledge the feelings of the campers involved.Adults ask campers what happened.When strong feelings are involved, staff help campers to respond appropriately.(e.g., staff encourage campers to brainstorm possible solutions, take time to "cool off", find an appropriate physical outlet)

Section III: Camper Choice, Planning & Reflection
Thinking in terms of being able to make choices, make plans, and reflect on your experiences, how frequently would you say the following occur: 0 to 10 I have opportunities to make individual or group plans for projects and activities.(e.g., written or sketched plan for a building project, verbal plans about an art project, or staff asks, "What is your plan?") There is a specific time or times for planning during the session routine.
I have opportunities to look back on things I am doing and make learning connections.
I have engaged in an intentional process of reflecting on what I am doing or have done.(e.g., writing in journals; sharing progress, accomplishments, or feelings) Activities involve structured times in which staff ask campers debrief questions.(e.g., questions that ask campers about the experiences they had in the activity) I have a say in how I spend my time at camp during activities.

Section IV: Learning at Camp
Thinking in terms of high expectations, good challenge, and working with others, how frequently would you say the following occur: 0 to 10 I am encouraged to try out new skills or attempt higher levels of performance.
Staff provide me with intentional opportunities for development of specific skills (as opposed to activities with just a recreation or 'having fun' focus).I am challenged (in a good way) by the activities.Activities are appropriately challenging (not too easy, not too hard) for me; I was not bored or frustrated.
There is sufficient time for all of the activities (e.g., I was not rushed, frustrated, bored, or distracted; I finished the activities).

I
have the opportunity to make at least one open-ended choice within activities.(e.g., campers decide roles, tools or materials, or topics) Staff share control of most activities with campers, providing guidance and facilitation while retaining overall responsibility.(e.g., staff use youth leaders, small groups, or individually guided activities.)

Table 1 . Camper Perceptions of Program Quality Subscale Descriptive Statistics CPQA Camper Survey subscale Mean Standard deviation Minimum Maximum Skewness
ReliabilityInternal consistency reliability was calculated for each subscale using Cronbach's alpha.This analysis was used to determine if the questions in each subscale measured the same general construct by producing similar scores.A Cronbach's alpha score of .7 to .8 is considered acceptable, and .8 to .9 is considered good.Cronbach's alpha scores for subscales in the survey ranged from a low of .843 to a high of .870.These results indicate that each subscale of questions in the CPQA Camper Survey can be considered reliable.Cronbach's alpha scores for each survey subscale are presented in Table2.

Table 2 . Reliability of Survey Subscales CPQA Camper Survey subscale Cronbach's alpha Number of items
The subscales of the CPQA Camper Survey were tested for correlations.Results indicate that all five subscales are significantly correlated to each other.This means that if a camper rated one program quality area high or low, they tended to score other quality areas in the same manner.

Table 4 . Percentage of Variance Explained by Factor Discussion
Think of the scale like … you begin, please think carefully about each of your answers.It is very important to be accurate, no answers are right or wrong, and everyone will have different answers.Please put down what you actually think for each question.Your answers will remain anonymous.Your responses are important to making camp better for future campers.By carefully completing this survey, you can help us make camp better.We want you to know that participation in this survey is voluntary.You can choose not to take part.You can choose not to finish the survey or skip any question you prefer not to answer without penalty.
DO YOU UNDERSTAND THE INSTRUCTIONS?IF NO, Please ask for clarification IF YES, Please turn the page and begin