Gauging Informal STEM Youth Program Impact: A Conceptual Framework and a Measurement Instrument

Robert H. Tai, Ji Hoon Ryoo, Claire E. Mitchell, Xiaoqing Kong, Angela Skeeles-Worley, John T. Almarode, Adam V. Maltese, Katherine P. Dabney


STEM education programs are often formulated with a "hands-on activities" focus across a wide array of topics from robotics to rockets to ecology. Traditionally, the impact of these programs is based on surveys of youth on program-specific experiences or the youths’ interest and impressions of science in general. In this manuscript, we offer a new approach to analyzing science programming design and youth participant impact. The conceptual framework discussed here concentrates on the organization and analysis of common learning activities and instructional strategies. We establish instrument validity and reliability through an analysis of validity threats and pilot study results. We conclude by using this instrument in an example analysis of a STEM education program.


activity-based science learning; informal STEM education; out-of-school time science education; informal STEM program impact

Full Text:



Adams, W. K., Perkins, K. K., Dubson, M., Finkelstein, N. D., & Wieman, C. E. (2004). The design and validation of the Colorado learning attitudes about science survey. Physics Education Research Conference.

Aschbacher, P. R., Li, E., & Roth, E. J. (2010). Is science me? High school students’ identities, participation, and aspiration in science, engineering, and medicine. Journal of Research in Science Teaching, 47(5), 564-582.

Bachman, N., Bischoff, P. J., Gallagher, H., Labroo, S., & Schaumloffel, J. C. (2008). PR2EPS: Preparation, recruitment, retention and excellence in the physical sciences, including engineering. A Report on the 2004, 2005 and 2006 science summer camps. Journal of STEM Education: Innovations & Research, 9(1/2), 30-39.

Barmby, P., Kind, P. M., & Jones, K. (2008). Examining changing attitudes in secondary school science. International Journal of Science Education, 30(8), 1075-1093.

Barman, C.R. (1999). Students’ views about scientists and school science: Engaging K-8 teachers in a national study. Journal of Science Teacher Education, 10(1), 43-54.

Barton, A. C., Kang, H., Tan, E., O'Neill, T. B., Bautista-Guerra, J., & Brecklin, C. (2013). Crafting a future in science: Tracing middle school girls' identity work overtime and space. American Educational Research Journal, 50(1), 37-75.

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246.

Bhattacharyya, S., & Mead, T. P. (2011). The influence of science summer camp on African American high school students’ career choices. School Science & Mathematics, 111(7), 345-353.

Brown, T. A. (2006). Confirmatory factor analysis for applied research. Guilford Press.

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Sage.

Carnevale, A. P., Smith, N., & Strohl, J. (2010). Help wanted: Projections of jobs and education requirements through 2018. Georgetown University, Center on Education and the Workforce.

Chang, S., Yeung, Y., & Cheng, M. H. (2009). Ninth graders’ learning interests, life experiences and attitudes towards science and technology. Journal of Science Education and Technology, 18, 447-457.

Choi, J., Fiszdon, J. M., & Medalia, A. (2010). Expectancy-value theory in persistence of learning effects in schizophrenia: Role of task value and perceived competency. Schizophrenia Bulletin, 36(5), 957-965.

Coleman, J. S., & Fararo, T. J. (Eds.). (1992). Rational choice theory advocacy and critique. Sage.

Cooper, K. M., Ashley, M., & Brownell, S. E. (2017). A Bridge to active learning: A summer bridge program helps students maximize their active-learning experiences and the active-learning experiences of others. CBE – Life Science Education, 16(1).

Deitrich, F., & List, C. (2013). A reason-based theory of rational choice. Noûs. 47(1), 104-134.

Ebenezer, J.V., & Zoller, U. (1993). Grade 10 students’ perceptions of and attitudes toward science teaching and school science. Journal of Research in Science Teaching, 30(2), 175-186.

Eccles J. S., Adler, T. F., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J. L., & Midgley, C. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), Achievement and achievement motivation (pp. 75-146). W. H. Freeman.

Eccles, J. S., & Wigfield, A. (1995). In the mind of the achiever: The structure of adolescents’ academic achievement related beliefs and self-perceptions. Personality and Social Psychology Bulletin, 21, 215-225.

Elam, M. E., Donham, B. L., & Solomon, S. R. (2012). An engineering summer program for underrepresented students from rural school districts. Journal of STEM Education: Innovations and Research, 13(2), 35-44.

Farland-Smith, D. (2009). Exploring middle school girls’ science identities: Examining attitudes and perceptions of scientists when working “side-by-side” with scientists. School Science and Mathematics, 109(7), 415-427.

Ferreira, M.M., & Trudel, A.R. (2012). The impact of problem-based learning on student attitudes toward science, problem-solving skills, and sense of community in the classroom. Journal of Classroom Interaction, 47(1), 23-30.

Fields, D. (2009). What do students gain from a week at science camp? Youth perceptions and the design of an immersive, research-oriented astronomy camp. International Journal of Science Education, 31(2), 151-171.

Fraser, B. J. (1978). Development of a test of science-related attitudes. Science Education, 62, 509-515.

Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction (6th ed.). Longman Publishing.

George, R. (2006). A cross-domain analysis of change in students’ attitudes toward science and attitudes about the utility of science. International Journal of Science Education, 28(6), 571-589.

Germann, P. J. (1988). Development of the attitude toward science in school assessment and its use to investigate the relationship between science achievement and attitude toward science in school. Journal of Research in Science Teaching, 25(8), 689-703.

Gibson, C., & Chase, H. L. (2002). Longitudinal impact of an inquiry-based science program on middle school students’ attitudes toward science. Science Education, 86(50), 693-705.

Guzzetti, B. J., & Bang, E. (2011). The influence of literacy-based science instruction on adolescents’ interest, participation, and achievement in science. Literacy Research and Instruction, 50, 44-67.

Häussler, P., & Hoffmann, L. (2000). A curricular frame for physics education: Development, comparison with students’ interests, and impact on students’ achievement and self-concept. Science Education, 84(6), 689-705.<689::AID-SCE1>3.0.CO;2-L

Homans, G. C. (1961). Social behavior: Its elementary focus. Harcourt.

Hu, L.-t., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55.

Hulleman, C. S., & Harackiewicz, J. M. (2009). Promoting interest and performance in high school science classes. Science, 326, 1410-1412.

Johnson, R. C. (2011). Using summer research to attract pre-college underrepresented students to STEM fields. Council on Undergraduate Research Quarterly, 31(3), 7-15.

Kind, P. M., Jones, K., & Barmby, P. (2007). Developing attitudes toward science measures. International Journal of Science Education, 29(7), 871–893.

Koballa, Jr., T. R., & Glynn, S. M. (2007). Attitudinal and motivational constructs in science learning. In S. K. Abell & N. G. Lederman (Eds.), Handbook of research on science education (pp. 75-102). Lawrence Erlbaum Associates.

Krajcik, J., Czerniak, C., and Berger, C. (2003). Teaching children science: A project-based approach (2nd ed.). McGraw-Hill.

Lent, R. W., Brown, S. D., & Gore, P. A. (1997). Discriminant and predictive validity of academic self-concept, academic self-efficacy, and mathematics-specific self-efficacy. Journal of Counseling Psychology, 44, 307-315.

Little, T. D., Slegers, D. W., & Card, N. A. (2006). A non-arbitrary method of identifying and scaling latent variables in SEM and MACS models. Structural Equation Modeling, 13, 59-72.

Loveland, M. T. (2003). Religious switching: Preference development, maintenance, and change. Journal for the Scientific Study of Religion, 42(1), 147-157.

Luce, M.R., & Hsi, S. (2014). Science-relevant curiosity expression and interest in science: An exploratory study. Science Education, 99(1), 70-97.

Machemer, P. L., & Crawford, P. (2007). Student perceptions of active learning in a large cross-disciplinary classroom. Active Learning in Higher Education, 8(1), 9-30.

Majumdar, S. K., Rosenfeld, L. M., Rubba, P. A., Miller, E. W., & Schmalz, R. F. (Eds.). (1991). Science education in the United States: Issues, crises, and priorities (pp. 92-105). The Pennsylvania Academy of Science.

Messick, S. (1994). Foundations of validity: Meaning and consequences in psychological assessment. European Journal of Psychological Assessment, 10(1), 1-9.

Moore, R. W., & Foy, R. (1997). The scientific attitude inventory: A revision (SAI II). Journal of Research in Science Teaching, 34(4), 327-36.<327::AID-TEA3>3.0.CO;2-T

Myers, R. E., & Fouts, J. T. (1992). A cluster analysis of high school science classroom environments and attitude toward science. Journal of Research in Science Teaching, 29(9), 929-937.

National Academy of Sciences. (2007). Rising above the gathering storm: Energizing and employing America for a brighter future. The National Academies Press.

Neter, J., Wasserman, W., & Kutner, M. H. (1989). Applied linear regression models (2nd ed.). Irwin.

Newell, A.D., Zientek, L.R., Tharp, B.Z., Vogt, G.L, & Moreno, N.P. (2015). Students’ attitudes toward science as predictors of gains on student content knowledge: Benefits of an after-school program. School Science and Mathematics, 115(5), 216-225.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.

Opp, K. (2019). The rationality of political protest: A comparative analysis of rational choice theory. Routledge.

Osborne, J., Simon, S., & Collin, S. (2003). Attitudes towards science: A review of the literature and its implications. International Journal of Science Education, 25(9), 1049-1079.

Robbins, M. E., Schoenfisch, M. H., Moore, J. T., & Tolar, D. (2005). An interactive analytical chemistry summer camp for middle school girls. Journal of Chemical Education, 82(10), 1486-1488.

Schiefele, U. (1991). Interest, learning, and motivation. Educational Psychologist, 26(3 & 4), 299-323.

Schoor, C., & Bannert, M. (2011). Motivation in a computer-supported collaborative learning scenario and its impact on learning activities and knowledge acquisition. Learning and Instruction, 21(4), 560-573.

Scott, J. (2000). Rational choice theory. In G. Browning, A. Halcli, & F. Webster (Eds.), Understanding contemporary society: Theories of the present (pp. 126-138). Sage.

Simpson, R. D., Koballa, Jr., T. R., Oliver, J. S., Crawley, F. E. (1994). Research on the affective dimension of science learning. In D. Gabel (Ed.) Handbook of research on science teaching and learning (pp. 211–234). Macmillan.

Sherkat, W. E., & Wilson, J. (1995). Preferences, constraints, and choices in religious markets: An examination of religious switching and apostasy. Social Forces, 73(3), 993-1026.

Swarat, S., Ortony, A., & Revelle, W. (2012). Activity matters: Understanding student interest in school science. Journal of Research in Science Teaching, 49(4), 515-537.

Talisayon, V. M., Mulemwa, J., Corrigan, D., Mehta, J., Jenkins, E., Koulaidis, V., Goel, V., Aikenhead, G., & Ogawa, M. (2004). Relevance of science education.

Talton, E. L., & Simpson, R. D. (1986). Relationships of attitude towards self, family, and school with attitude toward science among adolescents. Science Education, 70(4), 365-374.

Unfried, A., Faber, M., Stanhope, D. S., & Wiebe, E. (2015). The development and validation of a measure of student attitudes toward science, technology, engineering, and math (S-STEM). Journal of Psychoeducational Assessment, 33(7) 622-639.

Weinburgh, M. H., & Steele, D. (2000). The modified attitudes toward science inventory: Developing an instrument to be used with fifth grade urban students. Journal of Women and Minorities in Science and Engineering, 6, 87-98.

Welch, A. G. (2010). Using the TOSRA to assess high school students’ attitudes toward science after competing in the FIRST robotics competition: An exploratory study. Eurasia Journal of Mathematics, Science & Technology Education, 6(3), 187-197.

Wigfield, A. (1994). Expectancy-value theory of achievement motivation: A developmental perspective. Educational Psychology Review, 6(1), 49-78.

Wittek, R. (2013). Rational choice theory. In Theory in social and cultural anthropology: An encyclopedia (pp. 686-689). Sage.


Copyright (c) 2021 Robert H. Tai, Ji Hoon Ryoo, Claire E. Mitchell, Xiaoqing Kong, Angela Skeeles-Worley, John T. Almarode, Adam V. Maltese, Katherine P. Dabney

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.