Examining Effects of a Multisite Youth Outreach Program: A Meta-analysis Approach


  • Weiling Li Evaluation and Learning Research Center, Purdue University
  • Martha Lindley McDavid Evaluation and Learning Research Center, Purdue University
  • Sandra F. San Miguel College of Veterinary Medicine, Purdue University
  • Loran Carleton Parker Evaluation and Learning Research Center, Purdue University




STEM outreach, meta-analysis, multisite evaluation, heterogeneity, program effects


This paper presents the application of a meta-analysis approach to the evaluation of youth-learning data from the nationally distributed This is How We “Role” program. The application of meta-analysis for examining the impact of other multisite youth programs encountering similar data analysis challenges is discussed. At each This is How We “Role” program site, university partners collected data to examine youth-participant learning. Data analysis from these unique sites was challenging as the approach had to accommodate the innate heterogeneity across sites due to differences in implementation, sample size, and learning context. The meta-analysis method revealed details of the underlying variation between sites that could be masked by typical regression approaches, estimated overall program effects, examined subgroups and identified heterogeneity across project sites. The results showed the This is How We “Role” program generally increased learning at each site and as a whole, even though the program effects varied across sites. This example demonstrates the utility of using the meta-analysis approach to similar multi-site youth development programs.


Banks, S., McHugo, G. J., Williams, V. F., Drake, R. E., & Shinn, M. (2002). A prospective meta-analytic approach in a multisite study of homelessness prevention. New Directions for Program Evaluation, 2002(94), 45-60. https://doi.org/10.1002/ev.50

Bax, L., Yu, L. M., Ikeda, N., Tsuruta, H., & Moons, K.G.M.(2006). Development and validation of MIX: comprehensive free software for meta-analysis of causal research data. BMC Medical Research Methodology, 6(1), 1-11. https://doi.org/10.1186/1471-2288-6-50

Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley.

Borenstein (2019). Common mistakes in meta-analysis and how to avoid them. Biostat.

Chiolero, A., Santschi, V., Burnand, B., Platt, R. W., & Paradis, G. (2012). Meta-analyses: with confidence or prediction intervals? European Journal of Epidemiology, 27(10), 823-825. https://doi.org/10.1007/s10654-012-9738-y

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

Comprehensive Meta-Analysis (Version3.3.070) [Computer software]. Biostat Solutions.

Coryn, C. L. S., Hobson, K. A., & McCowen, R. H. (2015). Meta-analysis as a method of multi-site evaluation: An example from international development. Evaluation Journal of Australasia, 15(3), 4-14.

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8.

Lalkhen, A. G. (2008). Statistics V: Introduction to clinical trials and systematic reviews. Continuing Education in Anaesthesia Critical Care & Pain, 8 (4), 143-146. https://doi.org/10.1093/bjaceaccp/mkn023

Rosenthal R., DiMatteo, M .R. (2001). Meta-analysis: recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52(1), 59-82.

San Miguel, S., McDavid, L., Parker, L., & Simons, M. (2019). Developing a scalable STEM career development program for elementary school-aged students. Journal of STEM Outreach 2(1), 1-10.

Straw, R. B., & Herrell, J. M. (2002). A framework for understanding and improving multisite evaluations. New Directions for Evaluation, 2002(94), 5-16. https://doi.org/10.1002/ev.47

Turpin, R. S., and Sinacore, J. M. (Eds.). (1991). Multisite evaluations. New Directions for Evaluation, 50. 1-112. Jossey-Bass.

What Works Clearinghouse. (2020). Standards handbook (Version 4.1). https://ies.ed.gov/ncee/wwc/Handbooks






Research & Evaluation Studies