Group Mapping in a 4-H Camp Mixed-Methods Evaluation

Authors

  • Vikram Koundinya University of California, Davis and University of California Cooperative Extension https://orcid.org/0000-0003-2353-6468
  • Jenna Klink University of Wisconsin, Madison, Division of Extension
  • Joanna Skluzacek University of Wisconsin, Madison, Division of Extension
  • Claire Barrett University of Wisconsin, Madison
  • Cristina Chiarella University of California, Davis

DOI:

https://doi.org/10.5195/jyd.2020.891

Keywords:

4-H camp, evaluation, mixed methods, systems thinking, complex systems

Abstract

A week-long residential 4-H Agri-science youth camp had the goal of increasing understanding of agriculture as a system, however evaluating such a complex construct is difficult by adopting any one data-collection method. This paper presents the methods used in the evaluation of the camp and how these methods enhanced the overall program evaluation. The camp introduced 28 students, who were about 12 years of age from rural and urban counties of Wisconsin to the different aspects of the agricultural system, and promoted understanding of the connections among the various system components. The study utilized a mixed-methods triangulation design with group mapping (qualitative and quantitative), participant observation (qualitative), student reflections in the form of group presentations (qualitative), and retrospective post-then-pre surveys (quantitative) to measure and assess the anticipated outcomes. The qualitative assessment of the maps from Days 1 to 5 showed a significant improvement in students’ understandings of the interconnections of farm systems, which was validated by quantitative assessment of maps. Participant observation revealed that the camp provided students with a rich experiential learning experience and identified areas of improvement such as coordination of efforts between the different camp instructors. Student reflections and the end-of-camp survey indicated that the students increased their knowledge and skill levels as a result of attending the camp. The mixed-methods design used in this study serves as an example for designing better quality youth and Extension education program evaluations, as well as programs where the goal is to increase participants’ understandings of complex systems.

Author Biography

Vikram Koundinya, University of California, Davis and University of California Cooperative Extension

Evaluation Specialist

University of California (UC)-Daivs and UC Cooperative Extension

References

Arnold, M. E., & Cater, M. (2011). From then to now: Emerging directions for youth program evaluation. Journal of Youth Development, 6(3). 83-94. https://doi.org/10.5195/jyd.2011.176

Bell, A. (2007). Designing and testing questionnaires for children. Journal of Research in Nursing, 12(5), 461-469. https://doi.org/10.1177/1744987107079616

Bennet, T. (2018). Including the youth perspective: The development of the CPQA Camper Survey. Journal of Youth Development, 13(1-2), 266-285. https://doi.org/10.5195/jyd.2018.559

Besterfield-Sacre, M., Gerchak, J., Lyons, M., Shuman, L. J, & Wolfe, H. (2004). Scoring concept maps: An integrated rubric for assessing engineering education. Journal of Engineering Education, 93, 105-115. https://doi.org/10.1002/j.2168-9830.2004.tb00795.x

Brady, B., & O’Regan C. (2009). Meeting the challenge of doing an RCT evaluation of youth mentoring in Ireland: A journey in mixed methods. Journal of Mixed Methods Research, 3(3), 265-280. https://doi.org/10.1177/1558689809335973

Brown, J. S. (2006). Rethinking concept mapping for youth participatory evaluation in the context of youth development programs. Journal of Youth Development, 1(2). https://doi.org/10.5195/jyd.2006.388

Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6, 97-113. https://doi.org/10.1177/1468794106058877

Carlson, S. P., Heimlich, J. E., Storksdieck, M. & Meyer, N. (2009). Best practices for field days assessment tool and observation protocol. https://conservancy.umn.edu/handle/11299/172450

Creswell, J. W. & Plano Clark, V. L. (2006). Designing and conducting mixed methods research. Sage.

Dale, E. (1969). Audiovisual methods in teaching (3rd ed.). Dryden Press.

DeWalt, K. M. & DeWalt, B. R. (2011). Participant observation: A guide for fieldworkers (2nd ed.). Rowman & Littlefield.

Diem, K. G. (2002). Using research methods to evaluate your extension program. Journal of Extension, 40(6). https://www.joe.org/joe/2002december/a1.php

Edwards, H. C., Culp III, K., & Jordan, J. W. (2019). Using an innovative multiple-methods approach to evaluate extension conferences. Journal of Extension, 57(2). https://joe.org/joe/2019april/a1.php

Emery, M., Higgins, L., Chazdon, S., & Hansen, D. (2015). Using ripple effect mapping to evaluate program impact: Choosing or combining the methods that work best for you. Journal of Extension, 53(2). https://www.joe.org/joe/2015april/tt1.php

Fleiss, J. L., Levin, B., & Paik, M. C. (2003). Statistical methods for rates and proportions (3rd ed.). John Wiley & Sons.

Garton, M. S., Miltenberger, M., & Pruett, B. (2007). Does 4-H camp influence life skill and leadership development? Journal of Extension, 45(4). https://www.joe.org/joe/2007august/a4.php

Henderson, K., Powell, G., & Scanlin, M. (2005). Observing outcomes in youth development: An analysis of mixed methods. The Journal of Park and Recreation Administration, 3(4), 58-77. https://js.sagamorepub.com/jpra/article/view/1421

Jayaratne, K. S. U. (2001). Agricultural extension educators’ perceptions regarding the teaching and learning processes as related to sustainable agriculture: Implications for agricultural extension education (Unpublished doctoral dissertation). Iowa State University.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26. https://doi.org/10.3102/0013189X033007014

Johnson, R. B., Onwuegbuize, A. J., & Turner, L. A. (2007). Towards a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112-133. https://doi.org/10.1177/1558689806298224

Johnson, V., Ronan, K., Johnston, D., & Peace, R. (2014). Evaluations of disaster education programs for children: A methodological review. International Journal of Disaster Risk Reduction, 9, 107-123. https://doi.org/10.1016/j.ijdrr.2014.04.001

Jones, K., Gwynn, E., & Teeter, A. (2019). Quantitative or qualitative: Selecting the right methodological approach for credible evidence. Journal of Human Sciences and Extension, 7(2), 61-87. https://www.jhseonline.com/article/view/826

Jorgensen, D. (2015). Participant observation. In R. Scott & S. Kosslyn (Eds.), Emerging trends in the social and behavioral sciences (pp. 1-15). John Wiley & Sons.

Klatt, J., & Taylor-Powell, E. (2005). Using the retrospective-post-then-pre design. Quick Tips #27. University of Wisconsin-Extension. https://fyi.extension.wisc.edu/programdevelopment/files/2016/04/Tipsheet27.pdf

Klink, J., Koundinya, V., Kies, K., Robinson, C., Rao, A., Berezowitz, C., Widhalm, M, & Prokopy, L. (2017). Enhancing interdisciplinary climate change work through comprehensive evaluation. Climate Risk Management, 15, 109-125. https://doi.org/10.1016/j.crm.2016.11.003

Koundinya, V., & Martin, R. A. (2011). Teaching methods and tools used in food safety extension education programs in the North Central Region of the United States. International Journal of Agricultural Management & Development, 1, 157-167. https://doi.org/10.22004/ag.econ.246118

Kwaw-Mensah, D. (2008). Perceptions of agricultural extension educators regarding livestock waste management education in the North Central Region (Unpublished doctoral dissertation). Iowa State University.

Lewis, K. M., Bird, M., Wilkins, T., Borba, J., Nathaniel, K., & Schoenfelder, E. (2018). Developing a common evaluation tool for camps. Journal of Youth Development, 13(1-2), 306-315. https://doi.org/10.5195/jyd.2018.539

Mertens, D., & Hesse-Beber, S. (2013). Mixed methods and credibility of evidence in evaluation. New Directions for Evaluation, 138, 5-13. https://doi.org/10.1002/ev.20053

Morgan, D. (2007). Paradigms lost and pragmatism regained. Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1), 48-76. https://doi.org/10.1177/2345678906292462

Nathaniel, K., & Kinsey, S. B. (2013). Contributions of youth engagement to the development of social capital through community mapping. Journal of Extension, 51(1). https://www.joe.org/joe/2013february/tt7.php

National 4-H History Preservation Program. (2019). 4-H motto, creed and pledge. http://4-hhistorypreservation.com/History/M-C-P/

Patton, M. Q. (2015). Evaluation in the field: The need for site visit standards. American Journal of Evaluation, 36(4), 444-460. https://doi.org/10.1177/1098214015600785

Rallis, S. F. (2015). When and how qualitative methods provide credible and actionable evidence: Reasoning with rigor, probity, and transparency. In S. I. Donaldson, C. A. Christie, & M. M. Mark (Eds.), Credible and actionable evidence: The foundation for rigorous and influential evaluations (2nd ed., pp. 137–156). Sage.

Reeves, K. T., Donaldson, J. L., Naylor, M., & LeBleu, L. (2017). Youths perceive some improvement in substance abuse prevention knowledge, skills, and assets from participation in 4-H Health Rocks! Journal of Extension, 55(4). https://www.joe.org/joe/2017august/rb9.php

Rennekamp, R. A., & Engle, M. (2008). A case study in organizational change: Evaluation in Cooperative Extension. In M. T. Braverman, M. Engle, M. E. Arnold, & R. A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension [Special issue], New Directions for Evaluation, 2008(120), 15-26. Jossey-Bass.

Roberts, G. (2006). A philosophical examination of experiential learning theory for agricultural educators. Journal of Agricultural Education, 47(1), 17-29. https://doi.org/10.5032/jae.2006.01017

Sami, A. (2016). Integrating quantitative and qualitative data in mixed methods research--Challenges and benefits. Journal of Education and Learning, 5(3). http://dx.doi.org/10.5539/jel.v5n3p288

Sheldon, J., & Hopkins, L. (2008). Supporting Success. Why and how to improve quality in after-school programs. The James Irvine Foundation. http://hdl.handle.net/10244/63

Stair, K. S., Blackburn, J. J., Bunch, J. C., Blanchard, L., Cater, M., & Fox, J. (2016). Perceptions and educational strategies of Louisiana agricultural education teachers when working with students with special needs. Journal of Youth Development, 11(1). http://dx.doi.org/10.5195/JYD.2016.433

Stair, K. S., Moore, G. E., Wilson, B., Croom, B., & Jayaratne, K. S. U. (2010). Identifying confidence levels and instructional strategies of high school agricultural education teachers when working with students with special needs. Journal of Agricultural Education, 51(2), 90-101. https://doi.org/10.5032/jae.2010.02090

Wells, M. S., & Arthur-Banning, S. G. (2009). Mapping out your success: Using mind maps to evaluate youth development programs. Journal of Youth Development, 4(2). https://doi.org/10.5195/jyd.2009.268

Whittington, A., & Garst, B. A. (2018). The role of camp in shaping college readiness and building a pathway to the future for camp alumni. Journal of Youth Development, 13(1-2), 105-125. https://doi.org/10.5195/jyd.2018.519

Yohalem, N., Granger, R., & Pittman, K. (2009). The quest for quality: Recent developments and future directions for the out-of-school-time field. New Directions for Youth Development, 121, 129-140. https://doi.org/10.1002/yd.300

Downloads

Published

2020-12-15

Issue

Vol. 15 No. 6 (2020)

Section

Research & Evaluation Studies

License

Authors who publish with this journal agree to the following terms:

  1. The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
  2. Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
  3. The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons Attribution 4.0 International License or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
    1. Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
    with the understanding that the above condition can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
  4. The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
  5. Authors are permitted and encouraged to post online a prepublication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work. Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
  6. Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
  7. The Author represents and warrants that:
    1. the Work is the Author’s original work;
    2. the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
    3. the Work is not pending review or under consideration by another publisher;
    4. the Work has not previously been published;
    5. the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
    6. the Work contains no libel, invasion of privacy, or other unlawful matter.
  8. The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.

Revised 7/16/2018. Revision Description: Removed outdated link.