Building Evaluation Capacity in Youth-Serving Organizations Through Evaluation Advisory Boards

Barry A. Garst, James Pann, Tiffany Berry, Gretchen Biesecker, Jason Spector, Michael Conn, Curtis Jones


Youth-serving organizations seek effective and cost-efficient solutions to build evidence and advance their impact. Some common challenges include choosing data systems or assessments, budgeting and planning for 3rd-party studies, and refining measurement and outcomes when programs expand or change. Evaluation advisory boards (EABs) are a low-cost solution to add evaluation capacity and can be mutually beneficial to both youth-serving organizations and evaluation experts. Previous research suggests that EABs may encourage meaningful use of data, support internal evaluators, and/or facilitate difficult conversations among stakeholders. However, there are very few examples of successful EABs in practice. This paper shares the perspectives of EAB members and organizational evaluation leaders from a large national after-school program, After-School All-Stars (ASAS), including (a) a description of the benefits of EABs, (b) how EABs may be especially helpful with the context of the COVID-19 pandemic, and (c) examples of youth-serving organizations’ EABs. The experiences and lessons learned by ASAS and its EAB are generalizable to other non-profit youth development programs. Recommendations for structuring EABs based on organizational goals are provided.


capacity-building; evaluation; evaluation advisory boards; program quality

Full Text:



Afterschool Alliance. (2020). Afterschool & summer in the time of COVID-19(blog).

Altarum Institute (2012). Assessing the evaluation capacity of large nonprofit organizations: A detailed account of the methods, findings, and lessons learned from the YMCA Evaluation Capacity Assessment.

Arnold, M. E., & Cater, M. (2011). From then to now: Emerging directions for youth program evaluation. Journal of Youth Development, 6(3), 80-92.

Arnold, M. E., Cater, M., & Braverman, M. T. (2016). Rethinking evaluation capacity in youth development programs. The changing landscape of youth work: Theory and practice for an evolving field, 193-209.

Arnold, M. E., & Rennekamp, R.A. (2020). A time like no other: 4-H youth development and COVID-19. Journal of Extension, 58(3), 1-3.

Arnold, M. E., & Silliman, B. (2017). From theory to practice: A critical review of positive youth development program frameworks. Journal of Youth Development, 12(2), 1-20.

Baizerman, M. L., Fink, A., & VeLure Roholt, R. (2012). From consilium to advice: A review of the evaluation and related literature on advisory structures and processes. In R. VeLure Roholt & M. L. Baizerman (Eds.), Evaluation advisory groups. New Directions for Evaluation, 136, 5-29.

Bamberger, M., Rugh, J., & Mabry, L. (2006). Chapter 3 – Not enough money: addressing budget constraints. In M. Bamberger, J. Rugh, & L. Mabry (Eds.), Real World Evaluation (pp. 51-67). Sage.

Batras, D., Duff, C., & Smith, B. J. (2016). Organizational change theory: Implications for health promotion practice. Health Promotion International, 31(1), 231-241.

Baughman, S., Boyd, H. H., & Franz, N. K. (2012). Non-formal educator use of evaluation results. Evaluation and Program Planning, 35(3), 329-336.

Berry, T., & Sloper, M. (2020). Building effective continuous quality improvement systems: The need for evaluative thinking about out-of-school time program quality. In C. Newhouse & C. Russell (Eds.). Measure, Use, Improve! Data Use in Out-of-School Time. Information Age Publishing.

BoardSource. (2016). Advisory councils: Nine keys to success.

BoardSource. (2017). Leading with intent: 2017 national index of nonprofit board practices.

Bourgeois, I., & Cousins, J. (2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34(3), 299-319.

Boys and Girls Clubs of America. (2020, March 19). Clubs offer critical care during COVID-19 (blog).

Buckley, J., Archibald, T., Hargraves, M., & Trochim, W. M. (2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36(3), 375-388.

Bundy, J., Pfarrer, M. D., Short, C. E., & Coombs, W. T. (2017). Crises and crisis management: Integration, interpretation, and research development. Journal of Management, 43(6), 1661-1692.

Cohen, B. B. (2012). Advisory groups for evaluations in diverse cultural groups, communities, and contexts. In R. VeLure Roholt & M. L. Baizerman (Eds.), Evaluation advisory groups. New Directions for Evaluation, 136, 49-65.

Ettekal, A. V., & Agans, J. P. (2020). Positive youth development through leisure: Confronting the COVID-19 pandemic. Journal of Youth Development, 15(2), 1-20.

Fine, P., Reichle, S., & Lord, K. (2020, June 8). Youth or consequences: Put youth at the center of COVID-19 recovery (blog). The Brookings Institution.

Grembowski, D. (2001). The practice of program evaluation. Sage.

Johnston-Goodstar, K. (2012). Decolonizing evaluation: The necessity of evaluation advisory groups in Indigenous evaluation. New Directions for Evaluation, 136, 109-117.

Keller, E. (2010). Cutting costs, keeping quality: Financing strategies for youth-serving organizations in a difficult economy. Finance Project.

Kirby, T. (2020). Evidence mounts on the disproportionate effect of COVID-19 on ethnic minorities. The Lancet Respiratory Medicine, 8(6), 547-548.

Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33(3), 307-338.

Laurencin, C. T., & McClinton, A. (2020). The COVID-19 pandemic: A call to action to identify and address racial and ethnic disparities. Journal of Racial and Ethnic Health Disparities, 7(3), 398-402.

Lee, J. (2020). Mental health effects of school closures during COVID-19. The Lancet Child & Adolescent Health, 4(6), 421.

Lovell, A., Anucha, U., Houwer, R., & Galley, A. (2016). Beyond measure? The state of evaluation and action in Ontario’s youth sector. Youth Research and Evaluation eXchange.

Maley, M., Agans, E., Rainone, N., Cope, M., & Eckenrode, J. (2016). The evidence for outcomes from youth participation in 4-H. Bronfenbrenner Center for Translational Research.

Mariama-Arthur, K. (2017). Why selecting the 'right' advisory board members should matter to your organization. Forbes Coaches Council.

Mattessich, P. W. (2012). Advisory committees in contract and grant-funded evaluation projects. New Directions for Evaluation, 136, 31-48.

National Financial Educators Council. (2019). Standards and curriculum advisory boards.

Ouellette, R. R., Goodman, A. C., Martinez-Pedraza, F., Moses, J. O., Cromer, K., Zhao, X., Pierre, J., & Frazier, S. L. (2020). A systematic review of organizational and workforce interventions to improve the culture and climate of youth-service settings. Administration and Policy in Mental Health and Mental Health Services Research, 1-15.

Outley, C. W., & Blyth, D. A. (2020). Race, antiracism, and youth development: From awareness to sustained action. Journal of Youth Development, 15(5), 1-15.

Patton, M. Q. (2018). A historical perspective on the evolution of evaluative thinking. In A. T. Vo & T. Archibald (Eds.), Evaluative thinking. New Directions for Evaluation, 158, 11-28.

Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29, 443-459.

Preskill, H., & Tzavaras, T. (2006). Reframing evaluation through appreciative inquiry. Sage.

Rennekamp, R. A., & Arnold, M. (2009). What progress, program evaluation: Reflections on a quarter-century of Extension evaluation practice. Journal of Extension, 47(3), Article 3COM1.

Rogers, E. M. (2003). Diffusion of innovations, 5th edition. Free Press.

Silliman, B., Crinion, P., & Archibald, T. (2016). Evaluation champions: What they need and where they fit in organizational learning. Journal of Human Sciences and Extension, 4(3).

U.S. Department of Education. (2008, April). Building an effective advisory committee (Fact Sheet No. 21). Mentoring Resource Center.

Vo, A. T., Schreiber, J. S., & Martin, A. (2018). Toward a conceptual understanding of evaluative thinking. New Directions for Evaluation, 2018(158), 29-47.

Wong, C.A., Ming, D., Maslow, G., & Gifford, E.J. (2020). Mitigating the impacts of the COVID-19 pandemic response on at-risk children. Pediatrics.


Copyright (c) 2021 Barry A. Garst, James Pann, Tiffany Berry, Gretchen Biesecker, Jason Spector, Michael Conn, Curtis Jones

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.