Examining Youth Camping Outcomes Across Multiple States : the National 4-H Camping Research Consortium ( NCRC )

The impact of residential camp participation is needed for camps focused on a variety of outcomes including education, summer fun, prevention, and youth development. One system, the Cooperative Extension Service, conducts 4-H residential camps in most states nationwide every year. These camps, though offering educational enhancement and fun activities, are focused on youth development, incorporating a framework called the essential elements of positive youth development. The National 4-H Camping Research Consortium (NCRC), a group of Extension specialists and county-level educators, designed and piloted assessment tools for 4-H camps that can be used at any camp that focuses on youth development. The camp context questionnaire measures three essential elements of youth development: relationship with a caring adult, self-determination and mastery, and safe and inclusive environments. The life skill questionnaire measures three life skills: accepting self and others, accomplishing goals, and taking responsibility. Logic models and evaluation guidelines help camp directors plan camps that work for youth.


Background
Measuring the developmental outcomes of camp experiences for youth has been a major focus of many organizations.The American Camp Association (ACA) published the results of a national study of youth outcomes entitled, "Youth Development Outcomes of the Camp Experience" (ACA, 2005) in partnership with ACA-accredited camps including not-for-profit organizations, religious denominations, youth-serving agencies, municipalities, and independent camp owners.Although ACA's study generated excellent information about the impacts of youth camping in general, additional information has been needed about the youth outcomes of specific types of camps such as those focused on educational experiences (e.g.religious, sports, science and technology, and environment), those focused on preventative or therapeutic activities, and those targeting youth development.
One organization that focuses primarily on youth development is the 4-H program.It is one of the largest national agency providers of youth camping, involving 448,918 youths in 2008 (USDA, 2008).The 4-H program is a part of the national Cooperative Extension system, housed at land-grant universities, and is funded largely by the United States Department of Agriculture (USDA).Residential 4-H camps incorporate a framework called the "essential elements of youth development" into all camp activities.These elements include positive relationships with caring adults, emotionally and physically safe environments, opportunities for self-determination and mastery, a sense of belonging in an inclusive environment, and opportunities to value and practice service to others (American Camp Association, 2006;Kress, 2005;National 4-H Collaboration for Youth Members, 1999).
The 4-H camping program is not alone in its approach to youth development; other groups use similar frameworks (American Camp Association, 2006;America's Promise, 2000;Gambone & Arbreton, 1997).Most 4-H camps also emphasize life skill development and use a model developed by the Iowa State University.These life skills are divided into eight categories: caring, giving, working, being, living, thinking, managing, and relating (Hendricks, 1998).Although 4-H camps include the usual camping educational activities, the emphasis is heavily on positive youth development.
Although many land-grant universities evaluate 4-H camping each summer-and a strong body of literature has evolved over the past decade reflecting these evaluations (Arnold, Bourdeau, & Nagele, 2005;Bird, Coutellier, Borba, Dixon, & Horowitz, 2010;Ferrari, & McNeely, 2007;Forsythe, Matysik, & Nelson, 2004;Garst & Bruce, 2003;Garton, Miltenberger, & Pruett, 2007), systematic evaluation across multiple states has been rare.To respond to this need, in 2002, a group of 4-H camping specialists, researchers, and evaluators created the National 4-H Camping Research Consortium (NCRC) to coordinate the resources of multiple state 4-H programs in order to more effectively evaluate the outcomes of 4-H camping on a national level.Because this task required a collaborative effort that would be beyond the capabilities of individual states, a consortium approach was warranted.The NCRC was established as a working group of the National Association of Extension 4-H Agents' (NAE4-HA) Camping and Environmental Education Taskforce, as approved at the annual conference in Norfolk, VA, in October 2002.Initially, the State 4-H Office in Virginia coordinated the working group.
The taskforce established goals to accomplish the following objectives: 1) To develop a consortium of camping and research/evaluation professionals who would work cooperatively to explore the outcomes of 4-H camping on a national level 2) To develop standardized instruments and procedures to assess the outcomes of 4-H camping on a national level 3) To complete multistate 4-H camp evaluations using standardized instruments and procedures 4) To disseminate evaluation results at national Extension conferences 5) To disseminate evaluation results in at least one nationally recognized and peerreviewed journal 6) To develop a long-term plan for collaboration and continuance of the consortium.
Funding for the project came from three sources: 1) the Army/4-H Youth Camping Project, 2) Virginia Cooperative Extension (State 4-H Office at Virginia Tech), and the 3) Cooperative State Research, Education and Extension Service (CSREES), United States Department of Agriculture, now called the National Institute for Food and Agriculture (NIFA).

Developing the Consortium
The National 4-H Camping Research Consortium (NCRC) had its first face-to-face meeting in 2005 in Washington, D. C. where planning began for the development of standardized measures and procedures.The seven universities participating in the NCRC, were University of Maryland, Montana State University, University of Nebraska-Lincoln, The Ohio State University, Rutgers University, Virginia Tech, and West Virginia University.Support for the NCRC was provided by the 4-H National Headquarters through the involvement of Dr. Suzanne LeMenestrel.
The NCRC also developed important guidelines for partners working on a national Extensionwide project.Those guidelines included: 1) establish professional expectations with regard to youth development, evaluation and research, and Extension/USDA work; 2) be sensitive to university expectations with regard to promotion and tenure, research compliance, finance, and time away from work; 3) distribute work fairly according to individual differences in abilities and interests; 4) allow individuals to "buy-in" and "buy-out"; 5) establish dimensions of communication, including giving and accepting constructive criticism; 6) allow time for group formation and cohesion; 7) adhere to timelines and deadlines; and 8) recognize the contributions of all members.

Developing Standardized Instruments and Procedures
The result of the work of the NCRC was the creation of the National 4-H Camp Evaluation Tool Kit, which includes three logic models (one each for overall camp context of camping -essential elements of youth development, and life skills developed at camp); recommended practices for 4-H camp evaluation; a 4-H camp context survey instrument and a 4-H camp life skills survey instrument.
The development of each of the evaluation tools was built on two assumptions.The first assumption was that certain characteristics or features are necessary in youth programs in order for a particular experiential context to provide positive youth development.Based upon the work of the National 4-H Impact Design Implementation Team (National 4-H Headquarters, 1999), these eight essential elements are defined as: • a positive relationship with a caring adult • emotionally and physically safe environment • an inclusive environment • engagement in learning • opportunity for learning and mastery • opportunity to see oneself as an active participant in the future • opportunity for self-determination • opportunity to value and practice service for others.
These eight elements have been further synthesized into four core areas, identified as belonging, independence, mastery, and generosity (Kress, 2005).
The second assumption of the NCRC's program evaluation approach was that the goal of any 4-H camping experience should be to provide opportunities to practice life skills in a real-life setting such as camp.

Camp Logic Models
A logic model defines the intended outcomes to be experienced or achieved by program participants.It may be used for program planning, for program evaluation, or both.The three camping logic models developed by the NCRC provide a framework for describing the relationships among the investments made in camping programs, the activities or programs themselves, and the results or outcomes.They provide a common approach for integrating planning, implementation, evaluation, and reporting.These models are a guide for assessing camping programs.The NCRC logic models include: 1.These three logic models are not meant to be exhaustive, but rather are purposely general in nature so that the user may pick and choose the inputs, outputs, and outcomes that are relevant to his or her unique camping situation.Users may want to explore additional logic modeling resources for ways to expand or narrow the focus of the logic models in order to incorporate them into 4-H camp planning and evaluation efforts.
The 4-H Camp Context Questionnaire The "4-H Camp Context Questionnaire" was designed to measure whether a specific 4-H camp environment includes each of the eight essential elements throughout the course of the camp experience during a residential camp for youth ages 9-13.It uses a 4-point Likert scale with response categories 1 = strongly disagree, 2 = disagree, 3 = agree, 4 = strongly agree.
Although a camping program may choose to focus on one or more of the essential elements, it is highly recommended that the complete questionnaire be used.With this concept in mind, programs may choose to report only those elements selected for emphasis.Elements receiving less than favorable results may be targeted for future improvement.This instrument is meant to be descriptive.Camp directors will need to compare the results provided through this questionnaire with their camp mission, goals, and objectives.This comparison will assist in determining whether specific components of the camp program should be strengthened in order to increase the perceived presence of the elements.
The 4-H Camp Life Skill Questionnaire The "4-H Camp Life Skills Questionnaire" was designed to measure the acquisition of life skills during a residential 4-H camping program for youth ages 9-13.It uses a 4-point Likert scale with response categories 1 = strongly disagree, 2 = disagree, 3 = agree, 4 = strongly agree.Although a camping program may choose to focus on one or more of the life skills, it is recommended that the complete questionnaire be used.As with the camp context questionnaire, this instrument is also meant to be descriptive.Camp directors will need to use the results provided by this questionnaire, in comparison with their camp mission, goals, and objectives.This comparison will help to determine whether specific components of the camp program might develop life skills more effectively.

Recommended Practices for 4-H Camp Evaluation
The "recommended practices" for 4-H camp evaluation are meant to serve as a guide to Extension faculty, staff, and volunteers who engage in 4-H camp evaluation and research.In addition, these practices inform 4-H camp stakeholders about the practices that they should expect to be upheld by persons conducting 4-H camp evaluation and research.The recommended practices are meant to intentionally guide the decision-making processes involved in 4-H camp evaluation and research, and they draw heavily from the guiding principles and program standards of the American Evaluation Association (AEA, 2003).

Field Testing the Toolkit
In the summers of 2006 and 2007, several states piloted the camp evaluation instruments.Although most universities do not require institutional review board approval for pilot studies, each university was asked to follow its own rules.The primary purpose of the pilot tests was to improve the instrument and tool kit materials.In 2006, the context questionnaire, a 73-item questionnaire, and the life skills questionnaire, a 69-item questionnaire, were administered to more than 2,000 male and female youths ages 9-13 at the end of a residential camp experience across four states (Virginia, Ohio, Nebraska, and Kentucky).After that data were analyzed by statisticians at Virginia Tech, consortium members met to interpret the results as they pertained to the validity and reliability of the questionnaires.As a result, the questionnaires were shortened and refined and re-piloted in 2007.The new camp context questionnaire contained 33 items and the new life skills questionnaire contained 30 items.The life skill data was piloted in Virginia, West Virginia, and Kansas with 921 subjects, and the camp context data was piloted in Alaska, Montana, West Virginia, and Ohio with 1,016 subjects.Data were analyzed at the West Virginia University Extension Service.

Analysis and Results
In the 2007 pilot, factor analyses were conducted with the life skill data.Three life skills were identified: accepting self and others, accomplishing goals, and taking responsibility.Reliability and validity analyses were also performed.The questionnaire was again adjusted to reflect the final analysis.The life skills questionnaire now contains 26 items.It uses a four-point response scale: strongly disagree, disagree, agree, strongly agree.Table 1 lists the life skills identified in the second pilot test with corresponding questions and Cronbach's alpha scores.
Table 1 Life Skills at Camp -Questions and Cronbach's Alpha Scores Alpha Accepting Self and Others .8480 I was proud of my camp groups.I respected others who were different than me.I accepted people who thought or acted differently.I learned that some decisions are better than others.I learned that others' ideas were as important as mine.I made new friends.I respected other campers.I learned about my strengths and weaknesses.I treated others fairly.I was concerned about the well-being of others.I encouraged others to do their best.

Accomplishing Goals
.8631 I was proud of projects that I completed.I asked for help when I needed it.I chose to try new activities.I felt comfortable asking for help on a project.I always tried to do my best.I contributed to the success of the team.

Taking Responsibility
.7754 I was usually where I was supposed to be.I tried to do what was expected of me.I tried to solve problems without being violent.I was a good listener.I was responsible for my own behavior.I thought about a problem before trying to solve it.I thought carefully before making decisions.I tried to help if someone needed something.I cleaned up after myself.
Factor analyses were also conducted with the camp context data.Table 2 lists the results of the factor analyses and test for reliability in the second pilot test.Four elements of youth development were identified: opportunity to build a relationship with a caring adult, opportunity for self-determination and mastery, emotionally safe and inclusive environment, and physically safe environment.The camp context questionnaire now contains 30 items.Table 2 lists the life skills identified in the second pilot test with corresponding questions and Cronbach's alpha scores..8315My skills in some activities improved.My classes were interesting.I could make choices about how I spent my free time.I pushed myself to try harder because of challenging activities.I learned things that will be useful in the future.I felt like I had a choice in my camp classes.I felt good about something that I accomplished.I could make choices for recreation activities.Campers taught each other.Campers could be a part of making group decisions.Campers accomplished something they couldn't do the first day.Campers had the opportunity to learn about different careers.

Emotionally Safe and Inclusive Environment
.7737 Other kids did not like me.Other kids made fun of me.I was teased.Campers messed with others' belongings.Campers picked on one another.Mean jokes were played on campers.I felt free to express my opinion.I felt accepted by other campers.

Physically Safe Environment
.8477 I felt safe in my cottage/cabin.I felt safe at night.I felt safe in classes and activities.
A Case Study: Use of the questionnaires at West Virginia 4-H Camps Several states have adopted the use of the NCRC's logic models and youth questionnaires as a part of 4-H camp evaluation.In West Virginia, both the life skill and the camp context questionnaires have been implemented at county and state camps each year for the past three years.West Virginia 4-H camping data contains strong mean scores for each of the questions, and results of factor analyses show that the essential elements and life skills are consistent with the national data.However, further analysis of the camp context data for boys and girls has shown that boys do not score as high as girls on indicators of an emotionally safe and inclusive environment.Consistently, over three years, the analysis shows that boys indicate that they feel less emotionally safe at camp than girls.As a result of these findings, West Virginia University Extension 4-H educators and camp coordinators have made adjustments to the camp environment, including instituting mentoring and new-camper orientations, and have implemented stronger policies to prevent bullying.In the summer of 2010, focus groups will be held to discover the specific issues behind these findings.

Disseminating the Toolkit and Results
Members of the NCRC have disseminated their work to 4-H educators and other camp professionals in a variety of ways.Presentations were made to the American Evaluation Association, and to the annual conferences of the National Association of Extension 4-H Agents and the American Camp Association (ACA), among others.The logic models and the questionnaires have been shared at National 4-H Camping Institutes, the California 4-H Camping Conference, Mountaineer Camping Institute, and multiple section and regional ACA conferences.A tool kit containing the logic models, questionnaires, and recommended practices is available upon request (National Camping Research Consortium, 2007).

Future Plans and Projects
The NCRC will continue to develop other evaluation tools relevant to camp and 4-H youth development communities.Interests of consortium members include a questionnaire for teens to measure the impact of serving in a camp counselor role, questionnaires for camp staff, and qualitative methodologies such as focus group questions to better understand findings from the life skill and camp context instruments.

Benefits and Limitations of a Collaborative Evaluation Process
Some of the lessons about building an evaluation consortium that members of the NCRC learned include: • Reaching a consensus can take time.Therefore, collaborative planning for camp evaluation should begin well before the implementation of the camp.This supports the Extension programming model whereby purposeful planning for program outcomes occurs prior to the start of camp.
• When possible, other stakeholders, such as teen counselors, adult leaders and volunteers, summer camp staff members, parents, etc., should be allowed to provide input into the evaluation process.Again, this helps to generate buy-in.Support from these stakeholders is important, and recommended changes should be thoughtfully considered during the subsequent planning for camp evaluation.
• Survey design should allow each camp facility to add questions based upon individual needs.This flexibility encourages camp staff to care about the results and makes evaluation both relevant and responsive.
• Data should be collected from multiple sources (i.e., youths, leaders, and parents) to strengthen results and to explore different aspects of camping outcomes (i.e., immediate, short-term, and long-term impacts).
• Resources may limit what data can be collected.Camp faculty and staff need to recognize these limitations and structure research designs that are practical with given resources.

Conclusions
"High quality youth development doesn't just 'happen,' but rather it occurs through careful planning and the deliberate inclusion of certain elements" (Astroth, 1996).
The National Camping Research Consortium (NCRC) organized with the purpose of measuring whether those "certain elements" were present in the 4-H camping system.The evaluation tools they developed examine whether camps are providing the essential elements of youth development and are building life skills in youth.These instruments can be used in multistate or national settings, and aggregated data can give insights into the success and/or deficiencies of a camps or camp systems.The tools have been piloted and validated.The camp context questionnaire measures three essential elements of youth development: relationship with a caring adult, self-determination and mastery, and safe and inclusive environments.The life skill questionnaire measures three life skills: accepting self and others, accomplishing goals, and taking responsibility.Logic models and evaluation guidelines help camp directors plan camps that work for youth.A long-term plan for collaboration and continuance of the consortium is in place, including plans to develop tools to measure the impact of serving as a junior camp counselor or staff member.
Other agencies also use a youth development framework to design and implement their camping programs, even if their emphasis is on more specific educational or recreational goals.The evaluation tools contained in the National 4-H Camp Evaluation Tool Kit may be beneficial to many camps because there is growing consensus among youth development researchers, advocates and practitioners about the types of experiences that help develop a young person into a strong independent adult (Roth, 1998).
by the 4-H program nationwide.The essential elements are benchmarks for success in 4-H.3.The 4-H Camp Life Skills Outcomes Logic Model is a framework for measuring the life skill enhancement of campers.This logic model provides guidelines for inputs and outputs necessary to achieve life skill enhancement in youth.
The 4-H Camp Evaluation Logic Model is an overview for the evaluation of camping programs.The framework focuses on the work done at 4-H camps and highlights the outcomes that result from successful 4-H camping.These outcomes confirm the validity of 4-H camping and are the foundation for building a consistent 4-H camping program model nationwide.2.The 4-H CampContext Logic Model is a framework for understanding the camp environment based upon the eight essential elements of youth development developed and used

Table 2
Camp Context -Questions and Alpha Scores