Measuring STEM Learning in After-School Summer Programs: Review of the Literature

Authors

  • Erica Mallett Moore University of Washington
  • Ari Hock University of Washington
  • Bronwyn Bevan Wallace Foundation
  • Katie Headrick Taylor University of Washington

DOI:

https://doi.org/10.5195/jyd.2022.1131

Keywords:

out-of-school time, program evaluation, STEM learning, literature review

Abstract

After-school and summer programs provide important opportunities for youth to learn STEM practices and form STEM-related identities. However, there has been limited coordination across these programs to measure effectiveness toward learning outcomes. To better understand the constructs that are used to evaluate these programs, we searched key terms related to out-of-school time STEM learning on several education research databases. Our search uncovered 36 different tools. Across these tools, we discovered 76 measures, which were then grouped into 10 constructs based on similar themes. Constructs included: attitude toward science, career awareness and career interest, curiosity, engagement, home/school environment, interest, motivation, nature of science, self-efficacy, and STEM practices. Each construct is defined and clarified with examples from the tools. The review also considers tensions between attempts to standardize measures for evaluating program success and the need to account for equitable STEM learning pathways and adaptability across diverse communities.

References

Afterschool Alliance. (2012). Know your funders: A guide to STEM funding for afterschool. https://www.afterschoolalliance.org/STEM-Funding-Brief-10182012.pdf

Allen, P. J., Chang, R., Gorrall, B. K., Waggenspack, L., Fukuda, E., Little, T. D., & Noam, G. G. (2019). From quality to outcomes: A national study of afterschool STEM programming. International Journal of STEM Education, 6(1). https://doi.org/10.1186/s40594-019-0191-2

Azevedo, F. S. (2015). Sustaining interest-based participation in science. In K. A. Renninger, M. Nieswandt, & S. Hidi (Eds.) Interest in mathematics and science learning (pp. 281-296). American Educational Research Association. https://doi.org/10.3102/978-0-935302-42-4_16

Bandura, A. (1994). Self-efficacy. In V. S. Ramachaudran (Ed.), Encyclopedia of human behavior (Vol. 4, pp. 71-81). Academic Press. (Reprinted in H. Friedman [Ed.], Encyclopedia of mental health. Academic Press, 1998). https://www.uky.edu/~eushe2/Bandura/BanEncy.html

Bang, M., & Vossoughi, S. (2016). Participatory design research and educational justice: Studying learning and relations within social change making. Cognition and Instruction, 34(3), 173-193. http://dx.doi.org/10.1080/07370008.2016.1181879

Banks, J. A. (2006). Race, culture, and education: The selected works of James A. Banks. Routledge.

Barron, B., & Bell, P. (2015). Learning in informal and formal environments. In L. Corno & E. Anderman (Eds.), Handbook of educational psychology (3rd ed., ch. 24). Lawrence Erlbaum.

Barron, B. (2006). Interest and self-sustained learning as catalysts of development: A learning ecology perspective. Human Development, 49(4), 193-224. http://dx.doi.org/10.1159/000094368

Bell, J., Besley, J., Cannady, M., Crowley, K., Grack Nelson, A., Philips, T., Riedinger, K., & Storksdieck, M. (2019). The role of engagement in STEM learning and science communication: Reflections on interviews from the field. Center for Advancement of Informal Science Education.

Bevan, B. (2016). STEM learning ecologies: Relevant, responsive, and connected. Connected Science Learning, 1(1). NSTA Press. https://www.nsta.org/connected-science-learning/connected-science-learning-march-2016/stem-learning-ecologies

Bevan, B., Calabrese Barton A., & Garibay, C. (2018). Broadening perspectives on broadening participation in STEM: critical perspectives on the role of science engagement. Center for Advancement of Informal Science Education.

Blackley, S., & Howell, J. (2015). A STEM narrative: 15 years in the making. Australian Journal of Teacher Education, 40(7). http://dx.doi.org/10.14221/ajte.2015v40n7.8

Blosser, P. E. (1984). Attitude research in science education. Information Bulletin, No. 1. ERIC Clearinghouse of Science, Mathematics and Environmental Education. https://files.eric.ed.gov/fulltext/ED259941.pdf

Briggs, L., Trautmann, N., & Phillips, T. (2019). Exploring challenges and lessons learned in cross-cultural environmental education research. Evaluation and Program Planning. 73,156-162. https://doi.org/10.1016/j.evalprogplan.2019.01.001

Calabrese Barton, A., & Tan, E. (2010). We be burnin'! Agency, identity, and science learning. The Journal of the Learning Sciences, 19(2), 187-229. https://doi.org/10.1080/10508400903530044

Chalkiadaki, A. (2018). A systematic literature review of 21st century skills and competencies in primary education. International Journal of Instruction, 11(3), 1-16. https://doi.org/10.12973/iji.2018.1131a

Chen, S. (2006). Views on science and education (VOSE) questionnaire. Asia-Pacific Forum on Science Learning and Teaching, 7(2). https://www.eduhk.hk/apfslt/v7_issue2/chensf/index.htm

Collins, P. H. (2018). Confronting color-blind STEM talent development: Toward a contextual model for black student STEM identity. Journal of Advanced Academics, 29(2), 143-168. https://doi.org/10.1177/1932202X18757958

Collins, P. H., & Bilge, S. (2016). Intersectionality. Polity Press.

DiSalvo, C. (2012). Adversarial design (Design thinking, design theory). MIT Press.

Engeström, Y. (1999). Expansive visibilization of work: An activity-theoretical perspective. Computer Supported Cooperative Work (CSCW), 8(1), 63-93. https://link.springer.com/content/pdf/10.1023/A:1008648532192.pdf

Falk, H. J., Dierking, L. D., Babendure, J., Pattison, S., Meyer, D., Verbeke, M., Coe, M., Palmquist, S., Storksdieck, M., & Canzoneri, N. (2017). The learning SySTEM. STEM Ready America. http://stemreadyamerica.org/the-learning-system/

Flaskerud, J. H. (1988). Is the Likert scale format culturally biased?. Nursing Research, 37(3), 185–186. https://psycnet.apa.org/doi/10.1097/00006199-198805000-00013

Fu, A. C., Kannan, A., & Shavelson, R. J. (2019). Direct and unobtrusive measures of informal STEM education outcomes. New Directions for Evaluation, 2019(161), 35-57. https://doi.org/10.1002/ev.20348

Garibay, C., & Teasdale, R. M. (2019). Equity and evaluation in informal STEM education. In A. C. Fu, A. Kannan, & R. J. Shavelson (Eds.), Evaluation in informal science, technology, engineering, and mathematics education. New Directions for Evaluation, 161, 87-106. https://doi.org/10.1002/ev.20352

Germann P. J. (1988). Development of the attitude toward science in school assessment and its use to investigate the relationship between science achievement and attitude toward science in school. Journal of Research in Science Teaching, 25(8), 689-703. https://doi.org/10.1002/tea.3660250807

Glynn, S. M., Brickman, P., Armstrong, N., & Taasoobshirazi, G. (2011). Science motivation questionnaire II: Validation with science majors and nonscience majors. Journal of Research in Science Teaching, 48(10), 1159-1176. https://doi.org/10.1002/tea.20442

Grack Nelson, A., Goeke, M., Auster, R., Peterman, K., Lussenhop, A. (2019). Shared measures for evaluating common outcomes of informal STEM education experiences. New Directions for Evaluation, 161, 59-86. https://doi.org/10.1002/ev.20353

Greenberg, M. T., Domitrovich, C. E., Weissberg, R. P., & Durlak, J. A. (2017). Social and emotional learning as a public health approach to education. The Future of Children, 27(1), 13-32. https://www.jstor.org/stable/44219019

Halim, L., Abd Rahman, N., Zamri, R., & Mohtar, L. (2018). The roles of parents in cultivating children's interest towards science learning and careers. Kasetsart Journal of Social Sciences, 39(2), 190-196. https://so04.tci-thaijo.org/index.php/kjss/article/view/242487

Harty, H., & Beall, D. (1984). Toward the development of a children’s science curiosity measure. Journal of Research in Science Teaching, 21(4), 425-436. https://doi.org/10.1002/tea.3660210410

Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–127. https://doi.org/10.1207/s15326985ep4102_4

Hughes, R. (2015). An investigation into the longitudinal identity trajectories of women in science, technology, engineering, and mathematics. Journal of Women and Minorities in Science and Engineering, 21(3), 181-213. https://ui.adsabs.harvard.edu/link_gateway/2015JWMSE..21..181H/doi:10.1615/JWomenMinorScienEng.2015013035

Jenkins, H., Itō, M., & boyd, d. (2016). Participatory culture in a networked era: A conversation on youth, learning, commerce, and politics. Polity Press.

Joshi, A., Kale, S., Chandel, S., & Pal, D. (2015). Likert scale: Explored and explained. British Journal of Applied Science & Technology, 7(4), 396-403. http://dx.doi.org/10.9734/BJAST/2015/14975

Khishfe, R., & Abd-El-Khalick, F. (2002). Influence of explicit and reflective versus implicit inquiry-oriented instruction on sixth graders’ views of nature of science. Journal of Research in Science Teaching, 39(7), 551-578. https://doi.org/10.1002/tea.10036

Kpolovie, P. J., Joe, A. I., & Okoto, T. (2014). Academic achievement prediction: Role of interest in learning and attitude towards school. International Journal of Humanities Social Sciences and Education, 1(11), 73-100.

Lee, J. W., Jones, P. S., Mineyama, Y., & Zhang, X. E. (2002). Cultural differences in responses to a Likert scale. Research in Nursing & Health, 25(4), 295-306. https://doi.org/10.1002/nur.10041

Merriam-Webster. (n.d.). Framework. In Merriam-Webster.com dictionary. Retrieved June 25, 2020, from https://www.merriam-webster.com/dictionary/framework

Merriam-Webster. (n.d.). Instrument. In Merriam-Webster.com dictionary. Retrieved June 25, 2020, from https://www.merriam-webster.com/dictionary/instrument

Merriam-Webster. (n.d.). Mathematics. In Merriam-Webster.com dictionary. Retrieved June 25, 2020, from https://www.merriam-webster.com/dictionary/mathematics

Merriam-Webster. (n.d.). Measure. In Merriam-Webster.com dictionary. Retrieved June 25, 2020, from https://www.merriam-webster.com/dictionary/measure

Nasir, N. S. (2012). Racialized identities: Race and achievement among African American youth. Stanford University Press. https://doi.org/10.1515/9780804779142

Nasir, N., Rosebery, A. S., Warren, B., & Lee, C. D. (2006). Learning as a cultural process: Achieving equity through diversity. In K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (489-504). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.041

National Academies of Sciences, Engineering, and Medicine. (2018). How people learn II: Learners, contexts, and cultures. National Academies Press. https://doi.org/10.17226/24783

National Research Council. (2009). Learning science in informal environments: People, places, and pursuits. The National Academies Press. https://doi.org/10.17226/12190

National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. The National Academies Press. https://doi.org/10.17226/13165

National Research Council. (2015). Identifying and supporting productive STEM programs in out-of-school settings. Committee on successful out-of-school STEM learning. Board on Science Education, Division of Behavioral and Social Sciences and Education. The National Academies Press.

Noam, G. G., & Shah, A. (2014). Informal science and youth development: Creating convergence in out-of-school time. Teachers College Record, 116(13), 199-218.

Ogden, J., & Lo, J. (2012). How meaningful are data from Likert scales? An evaluation of how ratings are made and the role of the response shift in the socially disadvantaged. Journal of Health Psychology, 17(3), 350-361. https://doi.org/10.1177/1359105311417192

Organisation for Economic Co-operation and Development. (2006). OECD program for international student assessment 2006. https://nces.ed.gov/surveys/pisa/pdf/quest_pisa_2006_student.pdf

Philip, T. M., & Azevedo, F. S. (2017). Everyday science learning and equity: Mapping the contested terrain. Science Education, 101(4), 526-532. https://doi.org/10.1002/sce.21286

Saa Meroe, A. (2014). Democracy, meritocracy and the uses of education. The Journal of Negro Education, 83(4), 485. https://www.jstor.org/stable/10.7709/jnegroeducation.83.4.0485

Schiefele, U., Krapp, A., & Winteler, A. (1992). Interest as a predictor of academic achievement: A meta-analysis of research. In K. A. Renninger, S. Hidi, & A. Krapp (Eds.),The role of interest in learning and development (183-212). Erlbaum.

Schunk, D. H. (1991). Self-efficacy and academic motivation. Educational Psychologist, 26(3-4), 207-231. https://doi.org/10.1080/00461520.1991.9653133

Science Learning Activation Lab. (2016a). Scientific sensemaking. (Measures Technical Brief version 3.2). http://activationlab.org/wp-content/uploads/2018/03/Sensemaking-Report-3.2-20160331.pdf

Science Learning Activation Lab. (2016b). Engagement in science learning activities. (Measures Technical Brief version 3.2). http://activationlab.org/wp-content/uploads/2018/03/Engagement-Report-3.2-20160803.pdf

Sneider, C., & Noam, G. G. (2019). The common instrument suite: A means for assessing student attitudes in STEM classrooms and out-of-school environments. Connected Science Learning, 1(11). https://www.nsta.org/connected-science-learning/connected-science-learning-july-september-2019/common-instrument-suite

Subedi, B. P. (2016). Using Likert type data in social science research: Confusion, issues and challenges. International Journal of Contemporary Applied Sciences, 3(2), 36-49. http://www.ijcar.net/assets/pdf/Vol3-No2-February2016/02.pdf

Vakil, S. (2018). Ethics, identity, and political vision: Toward a justice-centered approach to equity in computer science education. Harvard Educational Review, 88(1), 26-52. https://doi.org/10.17763/1943-5045-88.1.26

Yerdelen-Damar, S., & Peşman, H. (2013). Relations of gender and socioeconomic status to physics through metacognition and self-efficacy, The Journal of Educational Research,106(4), 280-289. https://doi.org/10.1080/00220671.2012.692729

Downloads

Published

2022-06-15

Issue

Vol. 17 No. 2 (2022)

Section

Feature Articles

License

Authors who publish with this journal agree to the following terms:

  1. The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
  2. Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
  3. The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons Attribution 4.0 International License or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
    1. Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
    with the understanding that the above condition can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
  4. The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
  5. Authors are permitted and encouraged to post online a prepublication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work. Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
  6. Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
  7. The Author represents and warrants that:
    1. the Work is the Author’s original work;
    2. the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
    3. the Work is not pending review or under consideration by another publisher;
    4. the Work has not previously been published;
    5. the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
    6. the Work contains no libel, invasion of privacy, or other unlawful matter.
  8. The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.

Revised 7/16/2018. Revision Description: Removed outdated link.