Email this article 
Building Evaluation Capacity in Youth-Serving Organizations Through Evaluation Advisory Boards
Abstract
Youth-serving organizations seek effective and cost-efficient solutions to build evidence and advance their impact. Some common challenges include choosing data systems or assessments, budgeting and planning for 3rd-party studies, and refining measurement and outcomes when programs expand or change. Evaluation advisory boards (EABs) are a low-cost solution to add evaluation capacity and can be mutually beneficial to both youth-serving organizations and evaluation experts. Previous research suggests that EABs may encourage meaningful use of data, support internal evaluators, and/or facilitate difficult conversations among stakeholders. However, there are very few examples of successful EABs in practice. This paper shares the perspectives of EAB members and organizational evaluation leaders from a large national after-school program, After-School All-Stars (ASAS), including (a) a description of the benefits of EABs, (b) how EABs may be especially helpful with the context of the COVID-19 pandemic, and (c) examples of youth-serving organizations’ EABs. The experiences and lessons learned by ASAS and its EAB are generalizable to other non-profit youth development programs. Recommendations for structuring EABs based on organizational goals are provided.
Keywords
Full Text:
PDFReferences
Afterschool Alliance. (2020). Afterschool & summer in the time of COVID-19(blog). http://afterschoolalliance.org/covid
Altarum Institute (2012). Assessing the evaluation capacity of large nonprofit organizations: A detailed account of the methods, findings, and lessons learned from the YMCA Evaluation Capacity Assessment. http://www.evaluativethinking.org/docs/YMCA_Evaluation_Capacity_Assessment.pdf
Arnold, M. E., & Cater, M. (2011). From then to now: Emerging directions for youth program evaluation. Journal of Youth Development, 6(3), 80-92. https://doi.org/10.5195/jyd.2011.176
Arnold, M. E., Cater, M., & Braverman, M. T. (2016). Rethinking evaluation capacity in youth development programs. The changing landscape of youth work: Theory and practice for an evolving field, 193-209.
Arnold, M. E., & Rennekamp, R.A. (2020). A time like no other: 4-H youth development and COVID-19. Journal of Extension, 58(3), 1-3.
Arnold, M. E., & Silliman, B. (2017). From theory to practice: A critical review of positive youth development program frameworks. Journal of Youth Development, 12(2), 1-20. https://doi.org/10.5195/jyd.2017.17
Baizerman, M. L., Fink, A., & VeLure Roholt, R. (2012). From consilium to advice: A review of the evaluation and related literature on advisory structures and processes. In R. VeLure Roholt & M. L. Baizerman (Eds.), Evaluation advisory groups. New Directions for Evaluation, 136, 5-29.
Bamberger, M., Rugh, J., & Mabry, L. (2006). Chapter 3 – Not enough money: addressing budget constraints. In M. Bamberger, J. Rugh, & L. Mabry (Eds.), Real World Evaluation (pp. 51-67). Sage.
Batras, D., Duff, C., & Smith, B. J. (2016). Organizational change theory: Implications for health promotion practice. Health Promotion International, 31(1), 231-241. http://doi.org/10.1093/heapro/dau098
Baughman, S., Boyd, H. H., & Franz, N. K. (2012). Non-formal educator use of evaluation results. Evaluation and Program Planning, 35(3), 329-336. https://doi.org/10.1016/j.evalprogplan.2011.11.008
Berry, T., & Sloper, M. (2020). Building effective continuous quality improvement systems: The need for evaluative thinking about out-of-school time program quality. In C. Newhouse & C. Russell (Eds.). Measure, Use, Improve! Data Use in Out-of-School Time. Information Age Publishing.
BoardSource. (2016). Advisory councils: Nine keys to success. https://boardsource.org/wp-content/uploads/2017/01/9-Keys-Success-Advisory-Council.pdf?hsCtaTracking=8dacd2d1-12a2-4281-a6db-b5b03cca42de%7C57e67e8d-6a50-48a8-937c-96f7ffc7558d
BoardSource. (2017). Leading with intent: 2017 national index of nonprofit board practices. https://leadingwithintent.org/wp-content/uploads/2017/09/LWI2017.pdf
Bourgeois, I., & Cousins, J. (2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34(3), 299-319.
Boys and Girls Clubs of America. (2020, March 19). Clubs offer critical care during COVID-19 (blog). https://www.bgca.org/news-stories/2020/March/Clubs-Offer-Critical-Care-During-COVID-19
Buckley, J., Archibald, T., Hargraves, M., & Trochim, W. M. (2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36(3), 375-388. https://doi.org/10.1177/1098214015581706
Bundy, J., Pfarrer, M. D., Short, C. E., & Coombs, W. T. (2017). Crises and crisis management: Integration, interpretation, and research development. Journal of Management, 43(6), 1661-1692. https://doi.org/10.1177/0149206316680030
Cohen, B. B. (2012). Advisory groups for evaluations in diverse cultural groups, communities, and contexts. In R. VeLure Roholt & M. L. Baizerman (Eds.), Evaluation advisory groups. New Directions for Evaluation, 136, 49-65.
Ettekal, A. V., & Agans, J. P. (2020). Positive youth development through leisure: Confronting the COVID-19 pandemic. Journal of Youth Development, 15(2), 1-20. https://doi.org/10.5195/jyd.2020.962
Fine, P., Reichle, S., & Lord, K. (2020, June 8). Youth or consequences: Put youth at the center of COVID-19 recovery (blog). The Brookings Institution. https://www.brookings.edu/blog/future-development/2020/06/08/youth-or-consequences-put-youth-at-the-center-of-covid-19-recovery
Grembowski, D. (2001). The practice of program evaluation. Sage.
Johnston-Goodstar, K. (2012). Decolonizing evaluation: The necessity of evaluation advisory groups in Indigenous evaluation. New Directions for Evaluation, 136, 109-117. https://doi.org/10.1002/ev.20038
Keller, E. (2010). Cutting costs, keeping quality: Financing strategies for youth-serving organizations in a difficult economy. Finance Project. https://files.eric.ed.gov/fulltext/ED509450.pdf
Kirby, T. (2020). Evidence mounts on the disproportionate effect of COVID-19 on ethnic minorities. The Lancet Respiratory Medicine, 8(6), 547-548. https://doi.org/10.1016/S2213-2600(20)30228-9
Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33(3), 307-338. https://doi.org/10.1177/1098214011434608
Laurencin, C. T., & McClinton, A. (2020). The COVID-19 pandemic: A call to action to identify and address racial and ethnic disparities. Journal of Racial and Ethnic Health Disparities, 7(3), 398-402. https://doi.org/10.1007/s40615-020-00756-0
Lee, J. (2020). Mental health effects of school closures during COVID-19. The Lancet Child & Adolescent Health, 4(6), 421. https://doi.org/10.1016/S2352-4642(20)30109-7
Lovell, A., Anucha, U., Houwer, R., & Galley, A. (2016). Beyond measure? The state of evaluation and action in Ontario’s youth sector. Youth Research and Evaluation eXchange. https://yorkspace.library.yorku.ca/xmlui/bitstream/handle/10315/32819/YouthREX-BeyondMeasure-Report(1).pdf?sequence=1
Maley, M., Agans, E., Rainone, N., Cope, M., & Eckenrode, J. (2016). The evidence for outcomes from youth participation in 4-H. Bronfenbrenner Center for Translational Research. https://www.bctr.cornell.edu/wp-content/uploads/2016/12/Systemic-Translational-Review-4-H-evidence.pdf
Mariama-Arthur, K. (2017). Why selecting the 'right' advisory board members should matter to your organization. Forbes Coaches Council. https://www.forbes.com/sites/forbescoachescouncil/2017/12/11/why-selecting-the-right-advisory-board-members-should-matter-to-your-organization/#3892d7f76150
Mattessich, P. W. (2012). Advisory committees in contract and grant-funded evaluation projects. New Directions for Evaluation, 136, 31-48. https://doi.org/10.1002/ev.20032
National Financial Educators Council. (2019). Standards and curriculum advisory boards. https://www.financialeducatorscouncil.org/curriculum-advisory-board/
Ouellette, R. R., Goodman, A. C., Martinez-Pedraza, F., Moses, J. O., Cromer, K., Zhao, X., Pierre, J., & Frazier, S. L. (2020). A systematic review of organizational and workforce interventions to improve the culture and climate of youth-service settings. Administration and Policy in Mental Health and Mental Health Services Research, 1-15.
Outley, C. W., & Blyth, D. A. (2020). Race, antiracism, and youth development: From awareness to sustained action. Journal of Youth Development, 15(5), 1-15.
Patton, M. Q. (2018). A historical perspective on the evolution of evaluative thinking. In A. T. Vo & T. Archibald (Eds.), Evaluative thinking. New Directions for Evaluation, 158, 11-28.
Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29, 443-459. https://doi.org/10.1177/1098214008324182
Preskill, H., & Tzavaras, T. (2006). Reframing evaluation through appreciative inquiry. Sage.
Rennekamp, R. A., & Arnold, M. (2009). What progress, program evaluation: Reflections on a quarter-century of Extension evaluation practice. Journal of Extension, 47(3), Article 3COM1. https://archives.joe.org/joe/2009june/comm1.php
Rogers, E. M. (2003). Diffusion of innovations, 5th edition. Free Press.
Silliman, B., Crinion, P., & Archibald, T. (2016). Evaluation champions: What they need and where they fit in organizational learning. Journal of Human Sciences and Extension, 4(3).
U.S. Department of Education. (2008, April). Building an effective advisory committee (Fact Sheet No. 21). Mentoring Resource Center. https://educationnorthwest.org/sites/default/files/factsheet21.pdf
Vo, A. T., Schreiber, J. S., & Martin, A. (2018). Toward a conceptual understanding of evaluative thinking. New Directions for Evaluation, 2018(158), 29-47.
Wong, C.A., Ming, D., Maslow, G., & Gifford, E.J. (2020). Mitigating the impacts of the COVID-19 pandemic response on at-risk children. Pediatrics. https://doi.org/10.1542/peds.2020-0973
DOI: https://doi.org/10.5195/jyd.2021.1077
Copyright (c) 2021 Barry A. Garst, James Pann, Tiffany Berry, Gretchen Biesecker, Jason Spector, Michael Conn, Curtis Jones
This work is licensed under a Creative Commons Attribution 4.0 International License.
This journal is published by the University Library System, University of Pittsburgh as part of its D-Scribe Digital Publishing Program and is cosponsored by the University of Pittsburgh Press. The Journal of Youth Development is the official peer-reviewed publication of the National Association of Extension 4-H Youth Development Professionals and the National AfterSchool Association.
