Developing a Common Evaluation Tool for Camps

Evaluation has become a standard for youth programming, to provide both evidence for improvement recommendations and an assessment of program outcomes. Having a common evaluation tool across programs (in this case, camps) is beneficial in aggregating measurements and understanding similarities and differences between programs. The purpose of this paper is to describe the process of working with the California 4-H Camping Advisory Committee to develop a common evaluation tool for all California 4-H camps, and to share initial findings from the instrument. We present results from two years of data collection, and the multiple uses of the findings.


Introduction
Evaluation has become a standard for youth programming, to provide both evidence for improvement recommendations and an assessment of program outcomes.Having a common evaluation tool across programs (in this case, camps) is beneficial.Though programs often vary from location to location, similar programming, especially camps, share the same overall goals (Payne & McDonald, 2012).As such, the camps can benefit from utilizing the same evaluation tool.Common evaluation tools allow us to make data comparisons across local, state, and national programs.When developed properly, they ensure that all programs are using a valid and reliable evaluation measure.Indeed, several youth-serving organizations have developed common measures of key program outcomes, such as the National 4-H Common Measures; the Children, Youth, and Families At Risk (CYFAR) Program Common Measures; and the American Camp Association (ACA) Youth Outcomes Battery (YOB).
A challenge with common measures, however, is that each program has variations from other programs.Despite this challenge, the California (CA) 4-H Camping Advisory Committee decided to develop a common evaluation tool to be used in 4-H camps across the state.This paper describes the process used to create this tool and shares initial findings from the instrument.The first step of the process was to generate ideas about the key outcomes expected from CA 4-H camps.Recognizing the importance of understanding the camp experience from the participants' perspective, the evaluation coordinator met with the CA 4-H Camping Advisory Committee (which includes several of the authors of this paper) in fall 2015.This statewide committee is comprised of adult volunteers and teenagers who plan and participate in 4-H camps, program staff who oversee the volunteers, and academic staff.This discussion provided useful data to the evaluation team and began to lay a foundation of trust between the evaluators and camps.At the meeting, committee members were partnered to answer the question, "How does 4-H camp make a difference for kids?" and were encouraged to consider both campers and the teenage staff who plan and deliver the camp program.All ideas were listed and marked with an asterisk (*) if mentioned by more than one pair (see Table 1).definition of "relationship skills"?Knowing that we desired a short (no more than one page, double-sided) evaluation, we focused on four outcomes that the committee determined were most important across camps: affinity for nature, confidence, relationship skills, and responsibility.The group also chose to measure leadership skills for the teen staff (results for that measure can be found in Lewis, Bird, Borba, Nathaniel, & Schoenfelder, 2017).

Survey Development and Description
From there, the evaluation coordinator identified potential measures for each of these outcomes.Affinity for nature, relationship (or friendship) skills, and responsibility were all part of the ACA YOB.The session involved important stakeholders in a process similar to the one that the Camping Advisory Committee undertook of identifying outcomes for their camps and discussing the benefits of evaluation.Participants reviewed a draft of the proposed survey, offered feedback, and were asked to volunteer if their camp was interested in participating in the evaluation.To further develop buy-in and increase competence and confidence to deliver the tool, we held a webinar for camp staff that reviewed the purpose of the evaluation, the development of the tool, and general administration instructions.Finally, we developed a set of guidelines and a common "script" to be used to deliver the tool to youth, and answers to commonly asked questions (available from the first author).
We administered the survey at nine camps in summer 2016.Surveys were generally given out the last day or night before the youth left camp.Teens were encouraged to complete the survey at the same time to model the survey-taking to younger youth.Table 2 shows the demographics of youth and teens that completed the survey, as well as descriptive statistics for the outcomes.Table 3 presents the correlations among the outcomes.Following data analysis, camp staff from most of the camps in the evaluation attended a daylong retreat.The purpose of this gathering was to give camps the opportunity to explore the data, learn about the results of the data and discuss implications for their camps (i.e., were there changes that needed to be made to camp for the following year based on the results?).
As part of this process, the committee and camp staff discussed the outcomes that were assessed and felt that some of the questions did not fit the targeted outcomes.For example, the confidence subscale included the following statement: "I am satisfied with how I look."This was not relevant to the camp experience.Another example was the responsibility subscale; several items asked about making mistakes, owning up to mistakes, etc.While learning about one's mistakes is an important skill and relevant to the camp experience, the committee did not feel that these questions "fit" with what they had hoped to measure.As such, the group discussed why they initially wanted to measure responsibility and confidence, and why these measures weren't fitting.
For the 2017 evaluation, the committee decided to replace the confidence items with selfefficacy (see Lewis, 2015 for psychometric information) which focuses on youth's understanding of their own strengths (Lerner et al., 2005).These items are being collected from youth in the 4-H military program and were incorporated in the 4-H Social-Emotional Common Measure (Lewis, 2015), again allowing for cross-program comparisons.The committee decided to replace the responsibility scale with the independence scale (from the ACA YOB; psychometric information listed above).This scale, however, made references to being independent from one's family (e.g., "Being less dependent on my family").In consultation with the director of research from ACA, we changed this to "camp group" to better fit the camp program.Finally, many staff members mentioned the length of the friendship scale and asked to shorten it.Using exploratory factor analysis (EFA; Santos & Clegg, 1999), the data from the 2016 friendship skills scale was reduced from 14 items to seven items3 .
The Committee also expressed interest in learning more about whether campers felt safe and welcomed at camp.Thus, five items were added that assessed emotional safety (adapted from items piloted by the Thrive Foundation for Youth and YMCA).Variations of these items are also being collected in California from youth in the 4-H community club program, allowing for comparisons across the delivery modes in the state.Finally, an additional open-ended question was added to capture campers' most special moment at camp: "What was a special moment of camp?" We administered the revised survey at 12 camps in summer 2017.Five of these camps were returning from 2016.As with 2016, we held an informational webinar about how to administer the surveys and gave out the protocol.Table 4 shows the demographics of youth and teens that completed the survey in 2017, as well as descriptive statistics for the outcomes.Table 5 presents the correlations among the outcomes.

Discussion
This paper describes the process of developing and piloting a common evaluation tool across summer camps in the CA 4-H YDP.Having a common instrument has allowed the staff from the many 4-H camping programs to come together and discuss the results of the evaluation.These discussions gave camp staff the opportunity to become actively involved in the evaluation, experience the process of data analysis, and learn strategies to encourage reflection and utilization of findings.Without a common evaluation instrument, it would not have been possible to bring staff from the multiple camp programs together to discuss the findings and implications for their camps.The tool enabled comparisons of outcomes and experiences across camps with the composite data allowing the 4-H YDP to frame the findings statewide.Based on the state-level findings and individual camp findings, staff members were able to generate ideas for improvements for the following year and share ideas amongst themselves.

Limitations
This evaluation provided camps with valuable data that can be used to share the outcomes of the camp experience as well as to help camp staff improve programs for youth.It is not, however, without limitations.The evaluation is fairly short, allowing for only a small number of questions to be asked, and therefore does not capture all the potential outcomes of attending camps.
Each summer, the CA 4-H Youth Development Program (4-H YDP) runs approximately 20 summer resident camp programs across the state serving about 3,200 youth annually.4-H volunteers and youth plan and administer these local programs, with support from professional county-based staff.Most of the camps are 5 to 7 days long; they are teen-led and held at leased facilities.Camps are one of several program delivery modes 4-H embraces including 4-H clubs and after-school programming.In 2012 and 2013 the CA 4-H YDP collected data to measure outcomes for youth in 4-H resident camp programs.With the addition of an evaluation coordinator to the statewide 4-H staff in 2014, the process to evaluate and re-design a survey tool began.

Table 1 . 4-H Camping Advisory Committee Responses to the Question "How does 4-H camp make a difference for kids?" For teen Staff For both For campers
*Response was mentioned by more than one pair of committee members.A discussion was held about the outcomes to ensure that everyone was in agreement in regard to the operationalization of the outcomes discussed.For example, was there agreement on the Journal of Youth Development | http://jyd.pitt.edu/| Vol. 13 Issue 1-2 DOI 10.5195/jyd.2018.539Developing a Common Evaluation Tool for Camps 309

1
The committee decided to use the YOB, which enabled us to make comparisons to the national benchmark data from ACA.Additionally, we measured confidence using a subscale from the Positive Youth Development Inventory 2 (Arnold, Nott, & Meinhold).
This measure, also used with youth in the CA 4-H community club program, allows for comparisons across the delivery modes of CA 4-H programming.The tool we developed included two open-ended questions: "What was the best part of camp?" and "If you could change one thing about camp, what would make it better?"To engage camps, the two lead authors presented a workshop to volunteers and professional staff at the California 4-H Camping Conference in April 2016 prior to gathering data in summer.

Table 3 . Correlations among Survey Outcomes from Summer 2016 Affinity for nature Responsibility Friendship Confidence
***p < .001.