Measuring STEM Learning in After-School Summer Programs: Review of the Literature

Erica Mallett Moore, Ari Hock, Bronwyn Bevan, Katie Headrick Taylor

Abstract


After-school and summer programs provide important opportunities for youth to learn STEM practices and form STEM-related identities. However, there has been limited coordination across these programs to measure effectiveness toward learning outcomes. To better understand the constructs that are used to evaluate these programs, we searched key terms related to out-of-school time STEM learning on several education research databases. Our search uncovered 36 different tools. Across these tools, we discovered 76 measures, which were then grouped into 10 constructs based on similar themes. Constructs included: attitude toward science, career awareness and career interest, curiosity, engagement, home/school environment, interest, motivation, nature of science, self-efficacy, and STEM practices. Each construct is defined and clarified with examples from the tools. The review also considers tensions between attempts to standardize measures for evaluating program success and the need to account for equitable STEM learning pathways and adaptability across diverse communities.


Keywords


out-of-school time; program evaluation; STEM learning; literature review

Full Text:

PDF

References


Afterschool Alliance. (2012). Know your funders: A guide to STEM funding for afterschool. https://www.afterschoolalliance.org/STEM-Funding-Brief-10182012.pdf

Allen, P. J., Chang, R., Gorrall, B. K., Waggenspack, L., Fukuda, E., Little, T. D., & Noam, G. G. (2019). From quality to outcomes: A national study of afterschool STEM programming. International Journal of STEM Education, 6(1). https://doi.org/10.1186/s40594-019-0191-2

Azevedo, F. S. (2015). Sustaining interest-based participation in science. In K. A. Renninger, M. Nieswandt, & S. Hidi (Eds.) Interest in mathematics and science learning (pp. 281-296). American Educational Research Association. https://doi.org/10.3102/978-0-935302-42-4_16

Bandura, A. (1994). Self-efficacy. In V. S. Ramachaudran (Ed.), Encyclopedia of human behavior (Vol. 4, pp. 71-81). Academic Press. (Reprinted in H. Friedman [Ed.], Encyclopedia of mental health. Academic Press, 1998). https://www.uky.edu/~eushe2/Bandura/BanEncy.html

Bang, M., & Vossoughi, S. (2016). Participatory design research and educational justice: Studying learning and relations within social change making. Cognition and Instruction, 34(3), 173-193. http://dx.doi.org/10.1080/07370008.2016.1181879

Banks, J. A. (2006). Race, culture, and education: The selected works of James A. Banks. Routledge.

Barron, B., & Bell, P. (2015). Learning in informal and formal environments. In L. Corno & E. Anderman (Eds.), Handbook of educational psychology (3rd ed., ch. 24). Lawrence Erlbaum.

Barron, B. (2006). Interest and self-sustained learning as catalysts of development: A learning ecology perspective. Human Development, 49(4), 193-224. http://dx.doi.org/10.1159/000094368

Bell, J., Besley, J., Cannady, M., Crowley, K., Grack Nelson, A., Philips, T., Riedinger, K., & Storksdieck, M. (2019). The role of engagement in STEM learning and science communication: Reflections on interviews from the field. Center for Advancement of Informal Science Education.

Bevan, B. (2016). STEM learning ecologies: Relevant, responsive, and connected. Connected Science Learning, 1(1). NSTA Press. https://www.nsta.org/connected-science-learning/connected-science-learning-march-2016/stem-learning-ecologies

Bevan, B., Calabrese Barton A., & Garibay, C. (2018). Broadening perspectives on broadening participation in STEM: critical perspectives on the role of science engagement. Center for Advancement of Informal Science Education.

Blackley, S., & Howell, J. (2015). A STEM narrative: 15 years in the making. Australian Journal of Teacher Education, 40(7). http://dx.doi.org/10.14221/ajte.2015v40n7.8

Blosser, P. E. (1984). Attitude research in science education. Information Bulletin, No. 1. ERIC Clearinghouse of Science, Mathematics and Environmental Education. https://files.eric.ed.gov/fulltext/ED259941.pdf

Briggs, L., Trautmann, N., & Phillips, T. (2019). Exploring challenges and lessons learned in cross-cultural environmental education research. Evaluation and Program Planning. 73,156-162. https://doi.org/10.1016/j.evalprogplan.2019.01.001

Calabrese Barton, A., & Tan, E. (2010). We be burnin'! Agency, identity, and science learning. The Journal of the Learning Sciences, 19(2), 187-229. https://doi.org/10.1080/10508400903530044

Chalkiadaki, A. (2018). A systematic literature review of 21st century skills and competencies in primary education. International Journal of Instruction, 11(3), 1-16. https://doi.org/10.12973/iji.2018.1131a

Chen, S. (2006). Views on science and education (VOSE) questionnaire. Asia-Pacific Forum on Science Learning and Teaching, 7(2). https://www.eduhk.hk/apfslt/v7_issue2/chensf/index.htm

Collins, P. H. (2018). Confronting color-blind STEM talent development: Toward a contextual model for black student STEM identity. Journal of Advanced Academics, 29(2), 143-168. https://doi.org/10.1177/1932202X18757958

Collins, P. H., & Bilge, S. (2016). Intersectionality. Polity Press.

DiSalvo, C. (2012). Adversarial design (Design thinking, design theory). MIT Press.

Engeström, Y. (1999). Expansive visibilization of work: An activity-theoretical perspective. Computer Supported Cooperative Work (CSCW), 8(1), 63-93. https://link.springer.com/content/pdf/10.1023/A:1008648532192.pdf

Falk, H. J., Dierking, L. D., Babendure, J., Pattison, S., Meyer, D., Verbeke, M., Coe, M., Palmquist, S., Storksdieck, M., & Canzoneri, N. (2017). The learning SySTEM. STEM Ready America. http://stemreadyamerica.org/the-learning-system/

Flaskerud, J. H. (1988). Is the Likert scale format culturally biased?. Nursing Research, 37(3), 185–186. https://psycnet.apa.org/doi/10.1097/00006199-198805000-00013

Fu, A. C., Kannan, A., & Shavelson, R. J. (2019). Direct and unobtrusive measures of informal STEM education outcomes. New Directions for Evaluation, 2019(161), 35-57. https://doi.org/10.1002/ev.20348

Garibay, C., & Teasdale, R. M. (2019). Equity and evaluation in informal STEM education. In A. C. Fu, A. Kannan, & R. J. Shavelson (Eds.), Evaluation in informal science, technology, engineering, and mathematics education. New Directions for Evaluation, 161, 87-106. https://doi.org/10.1002/ev.20352

Germann P. J. (1988). Development of the attitude toward science in school assessment and its use to investigate the relationship between science achievement and attitude toward science in school. Journal of Research in Science Teaching, 25(8), 689-703. https://doi.org/10.1002/tea.3660250807

Glynn, S. M., Brickman, P., Armstrong, N., & Taasoobshirazi, G. (2011). Science motivation questionnaire II: Validation with science majors and nonscience majors. Journal of Research in Science Teaching, 48(10), 1159-1176. https://doi.org/10.1002/tea.20442

Grack Nelson, A., Goeke, M., Auster, R., Peterman, K., Lussenhop, A. (2019). Shared measures for evaluating common outcomes of informal STEM education experiences. New Directions for Evaluation, 161, 59-86. https://doi.org/10.1002/ev.20353

Greenberg, M. T., Domitrovich, C. E., Weissberg, R. P., & Durlak, J. A. (2017). Social and emotional learning as a public health approach to education. The Future of Children, 27(1), 13-32. https://www.jstor.org/stable/44219019

Halim, L., Abd Rahman, N., Zamri, R., & Mohtar, L. (2018). The roles of parents in cultivating children's interest towards science learning and careers. Kasetsart Journal of Social Sciences, 39(2), 190-196. https://so04.tci-thaijo.org/index.php/kjss/article/view/242487

Harty, H., & Beall, D. (1984). Toward the development of a children’s science curiosity measure. Journal of Research in Science Teaching, 21(4), 425-436. https://doi.org/10.1002/tea.3660210410

Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–127. https://doi.org/10.1207/s15326985ep4102_4

Hughes, R. (2015). An investigation into the longitudinal identity trajectories of women in science, technology, engineering, and mathematics. Journal of Women and Minorities in Science and Engineering, 21(3), 181-213. https://ui.adsabs.harvard.edu/link_gateway/2015JWMSE..21..181H/doi:10.1615/JWomenMinorScienEng.2015013035

Jenkins, H., Itō, M., & boyd, d. (2016). Participatory culture in a networked era: A conversation on youth, learning, commerce, and politics. Polity Press.

Joshi, A., Kale, S., Chandel, S., & Pal, D. (2015). Likert scale: Explored and explained. British Journal of Applied Science & Technology, 7(4), 396-403. http://dx.doi.org/10.9734/BJAST/2015/14975

Khishfe, R., & Abd-El-Khalick, F. (2002). Influence of explicit and reflective versus implicit inquiry-oriented instruction on sixth graders’ views of nature of science. Journal of Research in Science Teaching, 39(7), 551-578. https://doi.org/10.1002/tea.10036

Kpolovie, P. J., Joe, A. I., & Okoto, T. (2014). Academic achievement prediction: Role of interest in learning and attitude towards school. International Journal of Humanities Social Sciences and Education, 1(11), 73-100.

Lee, J. W., Jones, P. S., Mineyama, Y., & Zhang, X. E. (2002). Cultural differences in responses to a Likert scale. Research in Nursing & Health, 25(4), 295-306. https://doi.org/10.1002/nur.10041

Merriam-Webster. (n.d.). Framework. In Merriam-Webster.com dictionary. Retrieved June 25, 2020, from https://www.merriam-webster.com/dictionary/framework

Merriam-Webster. (n.d.). Instrument. In Merriam-Webster.com dictionary. Retrieved June 25, 2020, from https://www.merriam-webster.com/dictionary/instrument

Merriam-Webster. (n.d.). Mathematics. In Merriam-Webster.com dictionary. Retrieved June 25, 2020, from https://www.merriam-webster.com/dictionary/mathematics

Merriam-Webster. (n.d.). Measure. In Merriam-Webster.com dictionary. Retrieved June 25, 2020, from https://www.merriam-webster.com/dictionary/measure

Nasir, N. S. (2012). Racialized identities: Race and achievement among African American youth. Stanford University Press. https://doi.org/10.1515/9780804779142

Nasir, N., Rosebery, A. S., Warren, B., & Lee, C. D. (2006). Learning as a cultural process: Achieving equity through diversity. In K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (489-504). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.041

National Academies of Sciences, Engineering, and Medicine. (2018). How people learn II: Learners, contexts, and cultures. National Academies Press. https://doi.org/10.17226/24783

National Research Council. (2009). Learning science in informal environments: People, places, and pursuits. The National Academies Press. https://doi.org/10.17226/12190

National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. The National Academies Press. https://doi.org/10.17226/13165

National Research Council. (2015). Identifying and supporting productive STEM programs in out-of-school settings. Committee on successful out-of-school STEM learning. Board on Science Education, Division of Behavioral and Social Sciences and Education. The National Academies Press.

Noam, G. G., & Shah, A. (2014). Informal science and youth development: Creating convergence in out-of-school time. Teachers College Record, 116(13), 199-218.

Ogden, J., & Lo, J. (2012). How meaningful are data from Likert scales? An evaluation of how ratings are made and the role of the response shift in the socially disadvantaged. Journal of Health Psychology, 17(3), 350-361. https://doi.org/10.1177/1359105311417192

Organisation for Economic Co-operation and Development. (2006). OECD program for international student assessment 2006. https://nces.ed.gov/surveys/pisa/pdf/quest_pisa_2006_student.pdf

Philip, T. M., & Azevedo, F. S. (2017). Everyday science learning and equity: Mapping the contested terrain. Science Education, 101(4), 526-532. https://doi.org/10.1002/sce.21286

Saa Meroe, A. (2014). Democracy, meritocracy and the uses of education. The Journal of Negro Education, 83(4), 485. https://www.jstor.org/stable/10.7709/jnegroeducation.83.4.0485

Schiefele, U., Krapp, A., & Winteler, A. (1992). Interest as a predictor of academic achievement: A meta-analysis of research. In K. A. Renninger, S. Hidi, & A. Krapp (Eds.),The role of interest in learning and development (183-212). Erlbaum.

Schunk, D. H. (1991). Self-efficacy and academic motivation. Educational Psychologist, 26(3-4), 207-231. https://doi.org/10.1080/00461520.1991.9653133

Science Learning Activation Lab. (2016a). Scientific sensemaking. (Measures Technical Brief version 3.2). http://activationlab.org/wp-content/uploads/2018/03/Sensemaking-Report-3.2-20160331.pdf

Science Learning Activation Lab. (2016b). Engagement in science learning activities. (Measures Technical Brief version 3.2). http://activationlab.org/wp-content/uploads/2018/03/Engagement-Report-3.2-20160803.pdf

Sneider, C., & Noam, G. G. (2019). The common instrument suite: A means for assessing student attitudes in STEM classrooms and out-of-school environments. Connected Science Learning, 1(11). https://www.nsta.org/connected-science-learning/connected-science-learning-july-september-2019/common-instrument-suite

Subedi, B. P. (2016). Using Likert type data in social science research: Confusion, issues and challenges. International Journal of Contemporary Applied Sciences, 3(2), 36-49. http://www.ijcar.net/assets/pdf/Vol3-No2-February2016/02.pdf

Vakil, S. (2018). Ethics, identity, and political vision: Toward a justice-centered approach to equity in computer science education. Harvard Educational Review, 88(1), 26-52. https://doi.org/10.17763/1943-5045-88.1.26

Yerdelen-Damar, S., & Peşman, H. (2013). Relations of gender and socioeconomic status to physics through metacognition and self-efficacy, The Journal of Educational Research,106(4), 280-289. https://doi.org/10.1080/00220671.2012.692729




DOI: https://doi.org/10.5195/jyd.2022.1131

Copyright (c) 2022 Erica Mallett Moore, Ari Hock, Bronwyn Bevan, Katie Headrick Taylor

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.