Group Mapping in a 4-H Camp Mixed-Methods Evaluation

A week-long residential 4-H Agri-science youth camp had the goal of increasing understanding of agriculture as a system, however evaluating such a complex construct is difficult by adopting any one data-collection method. This paper presents the methods used in the evaluation of the camp and how these methods enhanced the overall program evaluation. The camp introduced 28 students, who were about 12 years of age from rural and urban counties of Wisconsin to the different aspects of the agricultural system, and promoted understanding of the connections among the various system components. The study utilized a mixed-methods triangulation design with group mapping (qualitative and quantitative), participant observation (qualitative), student reflections in the form of group presentations (qualitative), and retrospective post-then-pre surveys (quantitative) to measure and assess the anticipated outcomes. The qualitative assessment of the maps from Days 1 to 5 showed a significant improvement in students’ understandings of the interconnections of farm systems, which was validated by quantitative assessment of maps. Participant observation revealed that the camp provided students Journal of Youth Development | http://jyd.pitt.edu/ | Vol. 15 Issue 6 DOI 10.5195/jyd.2020.891 Mixed Methods 4-H Camp Evaluation 307 with a rich experiential learning experience and identified areas of improvement such as coordination of efforts between the different camp instructors. Student reflections and the end-of-camp survey indicated that the students increased their knowledge and skill levels as a result of attending the camp. The mixedmethods design used in this study serves as an example for designing better quality youth and Extension education program evaluations, as well as programs where the goal is to increase participants’ understandings of complex systems.


Introduction
Evaluation is a critical component of the youth development programming process as a means of gathering evidence needed for program improvement and outcomes-based assessment (Lewis et al., 2018). The evaluation of youth development programs has evolved over time from reporting outputs (e.g., number of participants) to measuring outcomes and assessing impact (Arnold & Cater, 2011;Rennekamp & Engle, 2008). While the predominant focus of youth development program evaluations continues to be measuring outcomes and assessing impact for accountability purposes, there has been a shift towards evaluating program quality (Arnold & Cater, 2011;Sheldon & Hopkins, 2008;Yohalem et al., 2009). This paradigm shift calls for the use of evaluation designs that can assess programs for improvement, outcome measurement and impact assessment purposes using both quantitative and qualitative methods.
Developing evaluative questions and selecting methods that can provide data to answer those questions is one of the first steps in evaluation (Rallis, 2015). If the data collection methods match the evaluation purpose, and are implemented correctly, evaluations will produce credible and actionable evidence (Rallis, 2015). Every method has unique advantages and inherent challenges, which need to be considered before selecting them for data collection (Jones et al., 2019). Relying on a single method can weaken the overall design as it is likely to be affected by the biases or weaknesses inherent in that method (R. Johnson et al., 2007). Mixed-methods designs can provide a more comprehensive account than either the qualitative or quantitative method used alone, offer a better explanation of the results, and enhance the integrity of the findings (Bryman, 2006).
For example, Edwards et al. (2019) found that adopting multiple methods like graffiti wall questions and sharing, reflective journaling, focus groups, roundtable discussions, interviews, storytelling, creative writing, and artistic renderings yielded richer data for evaluating their 4-H volunteer conference compared to quantitative surveys they previously utilized. Edwards and colleagues stated, "The 4-H emblem represents fourfold, holistic youth development. A quantitative assessment focuses on the head; however, it can overlook the impact on the heart, hands, and health of the individual. A mixed-methods evaluation strategy honors our holistic approach" (para 7). Klink et al. (2017) used mixed-methods design with qualitative and quantitative measures for evaluating an interdisciplinary climate change research and extension project and found that such a design enhanced the understanding of the project achievements and provided the ability to improve the project throughout its lifetime. This paper presents the methods used for evaluating a residential 4-H Agri-science youth camp and discusses how the use of mixed methods provided robust and comprehensive evaluative evidence related to program outcomes.

Research Question
What can we learn about the program, using group mapping and mixed methods?

4-H youth development educators and state Extension specialists at University of Wisconsin
(UW) collaborated with a UW-Platteville research facility to implement a week-long residential youth camp. Middle school-aged youth from 11 counties across Wisconsin attended the camp.
The team used a variety of teaching and learning methods such as lectures, discussions, small group experiential activities, group mapping exercises, and field visits. Field visits included visits to swine, poultry and dairy facilities on the farm; horticulture gardens; food sections of grocery stores; sunflower fields; and a UW irrigation and drainage water laboratory. Two evaluation specialists, a state 4-H specialist, and six 4-H Extension educators collaborated to develop the program. Program development happened through group meetings and individual work over the course of a year. In one of the planning meetings, the evaluation specialists helped facilitate a conversation where content specialists articulated the overall goal of the camp that all team members would work towards: for students to understand both agriculture as a system and the connections among the various system components. The evaluation was then designed around assessing that goal. Roberts' (2006) Model of the Experiential Learning Process and Edgar Dale's (1969) Cone of Experience were used in developing the educational curriculum and the pedagogical design of the camp. According to Roberts' experiential learning model, learning begins with an initial focus identified by the learner, which is followed by an initial experience, self-reflection on observations, and formulation of generalizations; this continues in a spiral-like pattern ( Figure   1). The teaching and learning methods of the camp were designed in a way that made the learning objectives for the camp clear to the students from the very beginning. We provided opportunities for both self-reflection about learning and peer observation, which help students generalize learning through the experiential learning process.

Figure 1. Experiential Learning Process (Roberts, 2006)
The educational curriculum and the teaching and learning processes centered on the 4-H educational philosophy of learning by doing (National 4-H History Preservation Program, 2019). Dale's (1969) Cone of Experience validates this educational philosophy, highlighting that people remember 90% of what they say and do. Several other researchers have found that experiential learning methods are effective in informal educational settings like Extension (Jayaratne, 2001;Koundinya & Martin, 2011;Kwaw-Mensah, 2008).

Study Approval and Design
The UW Extension Institutional Review Board (IRB) approved this study. The study utilized a mixed-methods triangulation design, where group mapping (qualitative and quantitative), participant observation (qualitative), student reflections in the form of group presentations (qualitative), and retrospective post-then-pre surveys (quantitative) were implemented during the camp with equal weight (Creswell & Plano Clark, 2006). The primary intended outcome of the camp was measuring the understanding of the systems concept on how the different components on a farm worked together. This required innovative methods and a mixedmethods triangulation design because it is difficult to comprehensively evaluate such constructs using quantitative or qualitative methods alone.

Participants
Twenty-eight students attended the camp. The average age of the students was 12 years, with almost 75% of them being sixth and seventh graders. A majority of the students (54%) were female and lived on a farm or visited one at least once a month. Students were from a mix of rural and urban counties.

Group Mapping
Students were divided into five groups and asked to draw their understanding of how the different components of the food system (e.g., plants, animals, humans, soils, water, technology, and energy) worked together as a system on a farm. Two 30-minute sessions were provided on the first 4 days of the camp towards this mapping exercise. Student groups used the same chart paper on all days of the camp and kept modifying their drawings as the camp progressed. A rubric with 11 evaluation criteria was developed by the team to rate the student group maps. The criteria were rated on a scale from 0 (Not at all) to 10 (Completely) to evaluate how well the different components were depicted on the maps. One evaluation specialist and the project director rated the maps every day using this rubric. Some students were involved in the rating process on Day 5 of the camp. Visual observation of the maps and mean scores were used to analyze the maps.
The 11 evaluation criteria on the rubric were rated by the evaluation specialist and the project director at the end of each day. Cohen's kappa coefficient of 0.417 was observed on Day 1 between the two raters. This score is considered as "fair" strength of association and should be treated as a matter of concern (Fleis et al., 2003). The raters compared their ratings from Day 1 and discussed where they differed and agreed upon criteria they would use to rate the maps on the remaining days. The scores of the two raters from the remaining days were consistent. Some students were involved in the rating process on Day 5 of the camp. They were asked to rate their own group map. Wherever they rated themselves low, the evaluation specialist discussed what elements needed to be added to get a higher score. This method served to reinforce student learning and as a participatory evaluation method.
Our group mapping method is similar to mind mapping, concept mapping, ripple effect mapping, and spider mapping or graphing methods that have been used to evaluate youth development and 4-H programs. Wells and Arthur-Banning (2009) stated that mind maps can be used to measure complex youth development concepts with relatively less money and time. Brown (2006) suggested that concept mapping helps with engaging youth in the research process with adults. The ripple effect mapping method, which employs a mind mapping approach, has been successfully used in Extension to document the impacts of 4-H youth programming (Emery et al., 2015). Similarly, an inquiry-based method called spider mapping or graphing has been used to understand the impact of youth engagement on community change work (Nathaniel & Kinsey, 2013).
Using rubrics to score maps has been found to be a useful method. Besterfield-Sacre et al.
(2004) used a rubric to evaluate the concept maps developed by industrial engineering students and found that this method helped them gain a deeper understanding of students' true conceptual knowledge and understanding.

Participant Observation
To carry out participant observation, the evaluation specialist attended the camp and used an observation guide, which allowed for focused and systematic observations (Carlson et al., 2009). Patton (2015) stated that understanding of programs is enhanced when observant evaluators attend the programs, pay attention to all details, and systematically document what they observe. Further, this method helps with investigating the rich, complex, problematic, and diverse experiences of people (Jorgensen, 2015). Dewalt and Dewalt (2011) stated that participant observation enhances the quality of the collected data and of the interpretations made after data analysis.

Student-Reflections
For the student-reflections method, each student group received a video camera at the beginning of the camp to document their learning in the form of photos or videos. The students that needed help with operating the video camera received assistance from the educator team.
The literature is replete with examples of the effectiveness of presentations as a learning method for students. This method has been identified by agricultural education teachers as one of the effective strategies for working with students with special needs (Stair et al., 2016;Stair et al., 2010).

End-of-Camp Survey
Students completed a paper survey on the last day which included questions related to the knowledge, skills, and understanding of the farm system component outcomes. Using a retrospective post-then-pre survey design (Klatt & Taylor-Powell, 2005), the survey consisted of both open-and closed-ended questions. Changes in knowledge and skills were analyzed using Likert-type items on a 5-point scale (scale points given in the results section). The survey also asked questions related to demographics and the social experiences of the students. Descriptive statistics were used to analyze the data. To ensure reliability, the survey was pilot tested with students attending other camps. The evaluation specialists carefully examined the pilot test data with the project team to ensure that the questions measured what we were intending to measure and would give us quality data. Pilot test results did not show any issues and no items were dropped from the surveys. 4-H county educators, a state 4-H specialist and the two evaluation specialists reviewed the survey for face and content validity.
Surveys are the most commonly used data collection methods in youth development and 4-H program evaluations for measuring and assessing different levels of outcomes. Bennet (2018) used camper self-report surveys to assess the quality of a summer camp and Whittington and Garst (2018) used retrospectively administered surveys to measure college readiness skills of youth camp participants. Surveys have been used in 4-H Extension programs to evaluate educational curricula (Reeves et al., 2017) and camps (Garton et al., 2007).

Results
The results are presented below according to the data-collection method utilized in the study and the research question: What can we learn about the program using group mapping and mixed methods?

Group Mapping
Visual observation of changes in the quality and complexity of maps from Days 1 to 5 showed a substantial improvement in students' understanding of systems concept, i.e., how the different components on a farm work together as a system. On Day 1 student groups drew little to nothing, whereas they were able to draw and depict interconnectedness between the different components like plants, farm machinery, animals, soil, rainfall and sun by Day 5 (Figure 2). Figure 2 can be found in the appendix.

Figure 2. Visual Depiction of Changes in Students' Understanding of Farm System Components Over Time Using Group Mapping
Mean scores for each group map were compared from Day 1 to Day 5 of camp, and mean score improvements of 3.65 to 6.30 on the 10-point scale were observed in all the five groups. These quantitative changes confirmed the qualitative observations of the group maps. For example, as the students improved the quality of the systems concepts in their map, the 11 criteria used to assess the maps quantitatively detected these changes as well.

Participant Observation
Observations indicated that students were engaged in the learning process and that they were forming new connections with each other and with the curriculum. It was also observed that the camp educators and other support staff were friendly with students and the entire atmosphere was conducive to learning. Enough opportunities and time were provided to reflect on what was being offered through the various teaching sessions, which is an essential component of the experiential learning cycle. The educators engaged students in question-andanswer sessions that furthered the reflection process. However, the time management during certain site visits was not very efficient. All the observation results were shared with the team.

End-of-Camp Survey
For clarity in presentation in Tables 1 and 2     Note. Likert-type scale for before attending the camp: 1 (No idea), 2 (Sounds a little familiar), 3 (Heard of it but couldn't really do it), 4 (I was basically able to do this), and 5 (Yes, I could definitely do this!). Likert-type scale for after attending the camp: 1 (No idea), 2 (Sounds a little familiar), 3 (Heard of it but can't really do it), 4 (I am basically able to do this), and 5 (Yes, I can definitely do this!).

Student-Reflections
Four groups gave presentations using PowerPoint including photos taken on different days of the camp, while one group developed and used a video to present their experiences. All the groups indicated that they gained an understanding of how the different systems worked together on a farm. The students commented on the many components of a farm system. One group stated, "We tested how soil would affect plants growing, and what it would do in water." They further indicated, "We planted seeds to see how they would grow without sun, water, soil, and fertilizer (seed, dirt, and FUN!!!)." The group that gave a video presentation indicated that working with telescopes and building windmills were the activities they enjoyed the most. The last group summarized their overall camp experience by saying " [We] really liked this camp so so much [and we] hope they make it next year." Three out of the five groups mentioned unprompted how the group mapping activity helped with their learning.

Discussion
We found how group mapping is useful to illustrate and understand students' learning about the agricultural system. Students also self-reported improvements in many knowledge and skill items on the survey, particularly the knowledge questions. Participant observation helped us understand how the learning environment contributed to students' knowledge and skills outcomes, engagement in learning, and areas for improvement. Student-reflections helped complete the story of learning and validated the systems-level learning the students illustrated in their maps.
Using mixed methods led to more robust evaluative evidence that investigated process along with outcomes and provided greater depth and breadth of information (Henderson et al., 2005;Sami, 2016). Brady and O'Regan (2009) found that adopting a mixed-methods design can also compensate for weaknesses of randomized control trials. In addition to the quantitative evidence, we got from rating the maps, we were able to examine the mental models (conceptualizations) students had of the farm as a system. In addition to allowing for the triangulation of data sources, each method was uniquely suited to measure a particular type of outcome or a different aspect of the overall camp goal. We found the group mapping method to be most suitable for assessing students' understandings of the overall farm systems concept, as such complex concepts were best suited for qualitative methods. Rating of the maps using a rubric provided an additional and different measure of changes in students' understanding of the concept of farm systems throughout the camp. The end-of-camp survey provided us with quantitative measures related to specific knowledge, skills, and overall camp experience outcomes. Qualitative measures used to capture these outcomes included participant observation and student reflections. Participant observation helped us to evaluate the process as well as the outcomes.
In addition to triangulating evaluation data from multiple sources and perspectives, the use of the group mapping method and student reflections about the process, and involving students in rating the maps contributed to student learning outcomes through reflection and discussion with peers. They were not viewed as separate evaluation activities but rather as integral to the pedagogical design of the camp and its learning outcomes. Summative student reflections and formative discussion and reflection throughout camp completed the experiential learning cycle for students.
There is growing agreement among youth development educators, 4-H educators, and evaluation specialists that surveys are among the least engaging methods for youth audiences, yet they are still one of the most often used methods (Bell, 2007;Diem, 2002;V. Johnson et al., 2014). Lack of resources and evaluation capacity could be one of the reasons for defaulting to surveys, which are usually inexpensive and easy to administer. The methods used in our study do not require large monetary resources, though they necessitate planning and collaboration with evaluation specialists. Youth program educators must think about evaluation design and methods during the initial program development stage and ideally engage evaluation specialists to design mixed methods that are uniquely suited to answer their specific research and evaluative questions. If evaluation specialists are not available, educators can use creative thinking to design methods that measure the overall goal(s) of their programming.
When it is not possible to use mixed methods, educators and evaluators should choose a method that is best suited to answer their program outcome goals, rather than defaulting to surveys (R. Johnson & Onwuegbuzie, 2004;V. Johnson et al., 2014;Mertens & Hesse-Beber, 2013;Morgan, 2007).

Limitations of the Study
The evaluation methods used in our study captured youth perspective (group mapping, student reflections, and surveys) and evaluator perspective (participant observation) but did not explicitly capture camp organizer or parent perspectives. Including those viewpoints in the evaluation would have made our study even more comprehensive. Also, our evaluation methods were conducted during the camp itself; we could not conduct follow-up evaluation. Following up with youth and/or their families to see how the camp experience and their learning resulted in any changes at home would also have made our study even more complete. Finally, the findings of this study have limited external validity. Achieving the outcomes is reliant on curriculum fidelity and place, thus the authors are not specifically validating the content, but the evaluation techniques used. However, the methods used have usability for other studies.

Implications for Future Research
In the future, researchers might study which types of data are most valued by different types of stakeholders. For example, in this study, the main project team of educators and specialists affirmed the value of our comprehensive approach, but in a future study, funders, youth, and parents could be consulted to interpret which methods were most worthwhile. Researchers might also study how proposed study designs differ when target audiences are involved in designing the evaluation as compared to when service providers design the evaluation. A unique feature of our study was the use of group mapping to understand student learning outcomes, but future studies could explore innovative methods to evaluate the program context and encompassing dynamics. For example, group mapping could include mapping the mental models plus identifying or illustrating which part of the experience led them to those increases in understanding. Finally, additional studies showcasing evaluation methods that enhance learning and engage youth will help further demonstrate the need and opportunity to use creative and diverse methods.
The methods presented in this paper can serve as examples to systematically develop and comprehensively evaluate youth, 4-H, and formal classroom programs. Multiple methods of evaluation create a stronger case for if and how program outcomes are achieved and strengthen the story for funders and stakeholders (Bryman, 2006;Klink et al., 2017).