Gauging Informal STEM Youth Program Impact: A Conceptual Framework and a Measurement Instrument

Authors

  • Robert H. Tai University of Virginia https://orcid.org/0000-0002-2804-2822
  • Ji Hoon Ryoo Yonsei University
  • Claire E. Mitchell Montana State University
  • Xiaoqing Kong University of Virginia
  • Angela Skeeles-Worley University of Virginia
  • John T. Almarode James Madison University
  • Adam V. Maltese Indiana University Bloomington
  • Katherine P. Dabney Virginia Commonwealth University

DOI:

https://doi.org/10.5195/jyd.2021.981

Keywords:

activity-based science learning, informal STEM education, out-of-school time science education, informal STEM program impact

Abstract

STEM education programs are often formulated with a "hands-on activities" focus across a wide array of topics from robotics to rockets to ecology. Traditionally, the impact of these programs is based on surveys of youth on program-specific experiences or the youths’ interest and impressions of science in general. In this manuscript, we offer a new approach to analyzing science programming design and youth participant impact. The conceptual framework discussed here concentrates on the organization and analysis of common learning activities and instructional strategies. We establish instrument validity and reliability through an analysis of validity threats and pilot study results. We conclude by using this instrument in an example analysis of a STEM education program.

Author Biography

Robert H. Tai, University of Virginia

Associate Professor of Science Education, University of Virginia School of Education and Human Development

References

Adams, W. K., Perkins, K. K., Dubson, M., Finkelstein, N. D., & Wieman, C. E. (2004). The design and validation of the Colorado learning attitudes about science survey. Physics Education Research Conference. http://www.colorado.edu/sei/class/

Aschbacher, P. R., Li, E., & Roth, E. J. (2010). Is science me? High school students’ identities, participation, and aspiration in science, engineering, and medicine. Journal of Research in Science Teaching, 47(5), 564-582. https://doi.org/10.1002/tea.20353

Bachman, N., Bischoff, P. J., Gallagher, H., Labroo, S., & Schaumloffel, J. C. (2008). PR2EPS: Preparation, recruitment, retention and excellence in the physical sciences, including engineering. A Report on the 2004, 2005 and 2006 science summer camps. Journal of STEM Education: Innovations & Research, 9(1/2), 30-39.

Barmby, P., Kind, P. M., & Jones, K. (2008). Examining changing attitudes in secondary school science. International Journal of Science Education, 30(8), 1075-1093. https://doi.org/10.1080/09500690701344966

Barman, C.R. (1999). Students’ views about scientists and school science: Engaging K-8 teachers in a national study. Journal of Science Teacher Education, 10(1), 43-54. http://www.jstor.org/stable/43156207

Barton, A. C., Kang, H., Tan, E., O'Neill, T. B., Bautista-Guerra, J., & Brecklin, C. (2013). Crafting a future in science: Tracing middle school girls' identity work overtime and space. American Educational Research Journal, 50(1), 37-75. https://doi.org/10.3102/0002831212458142

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246. https://doi.org/10.1037/0033-2909.107.2.238

Bhattacharyya, S., & Mead, T. P. (2011). The influence of science summer camp on African American high school students’ career choices. School Science & Mathematics, 111(7), 345-353. https://doi.org/10.1111/j.1949-8594.2011.00097.x

Brown, T. A. (2006). Confirmatory factor analysis for applied research. Guilford Press.

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Sage.

Carnevale, A. P., Smith, N., & Strohl, J. (2010). Help wanted: Projections of jobs and education requirements through 2018. Georgetown University, Center on Education and the Workforce.

Chang, S., Yeung, Y., & Cheng, M. H. (2009). Ninth graders’ learning interests, life experiences and attitudes towards science and technology. Journal of Science Education and Technology, 18, 447-457. https://doi.org/10.1007/s10956-009-9162-6

Choi, J., Fiszdon, J. M., & Medalia, A. (2010). Expectancy-value theory in persistence of learning effects in schizophrenia: Role of task value and perceived competency. Schizophrenia Bulletin, 36(5), 957-965. https://doi.org/10.1093/schbul/sbq078

Coleman, J. S., & Fararo, T. J. (Eds.). (1992). Rational choice theory advocacy and critique. Sage.

Cooper, K. M., Ashley, M., & Brownell, S. E. (2017). A Bridge to active learning: A summer bridge program helps students maximize their active-learning experiences and the active-learning experiences of others. CBE – Life Science Education, 16(1). https://doi.org/10.1187/cbe.16-05-0161

Deitrich, F., & List, C. (2013). A reason-based theory of rational choice. Noûs. 47(1), 104-134. https://doi.org/10.1111/j.1468-0068.2011.00840.x

Ebenezer, J.V., & Zoller, U. (1993). Grade 10 students’ perceptions of and attitudes toward science teaching and school science. Journal of Research in Science Teaching, 30(2), 175-186. https://doi.org/10.1002/tea.3660300205

Eccles J. S., Adler, T. F., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J. L., & Midgley, C. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), Achievement and achievement motivation (pp. 75-146). W. H. Freeman.

Eccles, J. S., & Wigfield, A. (1995). In the mind of the achiever: The structure of adolescents’ academic achievement related beliefs and self-perceptions. Personality and Social Psychology Bulletin, 21, 215-225. https://doi.org/10.1177/0146167295213003

Elam, M. E., Donham, B. L., & Solomon, S. R. (2012). An engineering summer program for underrepresented students from rural school districts. Journal of STEM Education: Innovations and Research, 13(2), 35-44.

Farland-Smith, D. (2009). Exploring middle school girls’ science identities: Examining attitudes and perceptions of scientists when working “side-by-side” with scientists. School Science and Mathematics, 109(7), 415-427. https://doi.org/10.1111/j.1949-8594.2009.tb17872.x

Ferreira, M.M., & Trudel, A.R. (2012). The impact of problem-based learning on student attitudes toward science, problem-solving skills, and sense of community in the classroom. Journal of Classroom Interaction, 47(1), 23-30. https://www.jstor.org/stable/43858871

Fields, D. (2009). What do students gain from a week at science camp? Youth perceptions and the design of an immersive, research-oriented astronomy camp. International Journal of Science Education, 31(2), 151-171. https://doi.org/10.1080/09500690701648291

Fraser, B. J. (1978). Development of a test of science-related attitudes. Science Education, 62, 509-515. https://doi.org/10.1002/sce.3730620411

Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction (6th ed.). Longman Publishing.

George, R. (2006). A cross-domain analysis of change in students’ attitudes toward science and attitudes about the utility of science. International Journal of Science Education, 28(6), 571-589. https://doi.org/10.1080/09500690500338755

Germann, P. J. (1988). Development of the attitude toward science in school assessment and its use to investigate the relationship between science achievement and attitude toward science in school. Journal of Research in Science Teaching, 25(8), 689-703. https://doi.org/10.1002/tea.3660250807

Gibson, C., & Chase, H. L. (2002). Longitudinal impact of an inquiry-based science program on middle school students’ attitudes toward science. Science Education, 86(50), 693-705. https://doi.org/10.1002/sce.10039

Guzzetti, B. J., & Bang, E. (2011). The influence of literacy-based science instruction on adolescents’ interest, participation, and achievement in science. Literacy Research and Instruction, 50, 44-67. https://doi.org/10.1080/19388070903447774

Häussler, P., & Hoffmann, L. (2000). A curricular frame for physics education: Development, comparison with students’ interests, and impact on students’ achievement and self-concept. Science Education, 84(6), 689-705. https://doi.org/10.1002/1098-237X(200011)84:6<689::AID-SCE1>3.0.CO;2-L

Homans, G. C. (1961). Social behavior: Its elementary focus. Harcourt.

Hu, L.-t., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55. https://doi.org/10.1080/10705519909540118

Hulleman, C. S., & Harackiewicz, J. M. (2009). Promoting interest and performance in high school science classes. Science, 326, 1410-1412. https://doi.org/10.1126/science.1177067

Johnson, R. C. (2011). Using summer research to attract pre-college underrepresented students to STEM fields. Council on Undergraduate Research Quarterly, 31(3), 7-15.

Kind, P. M., Jones, K., & Barmby, P. (2007). Developing attitudes toward science measures. International Journal of Science Education, 29(7), 871–893. https://doi.org/10.1080/09500690600909091

Koballa, Jr., T. R., & Glynn, S. M. (2007). Attitudinal and motivational constructs in science learning. In S. K. Abell & N. G. Lederman (Eds.), Handbook of research on science education (pp. 75-102). Lawrence Erlbaum Associates.

Krajcik, J., Czerniak, C., and Berger, C. (2003). Teaching children science: A project-based approach (2nd ed.). McGraw-Hill.

Lent, R. W., Brown, S. D., & Gore, P. A. (1997). Discriminant and predictive validity of academic self-concept, academic self-efficacy, and mathematics-specific self-efficacy. Journal of Counseling Psychology, 44, 307-315. https://doi.org/10.1037/0022-0167.44.3.307

Little, T. D., Slegers, D. W., & Card, N. A. (2006). A non-arbitrary method of identifying and scaling latent variables in SEM and MACS models. Structural Equation Modeling, 13, 59-72. https://doi.org/10.1207/s15328007sem1301_3

Loveland, M. T. (2003). Religious switching: Preference development, maintenance, and change. Journal for the Scientific Study of Religion, 42(1), 147-157. https://doi.org/10.1111/1468-5906.00168

Luce, M.R., & Hsi, S. (2014). Science-relevant curiosity expression and interest in science: An exploratory study. Science Education, 99(1), 70-97. https://doi.org/10.1002/sce.21144

Machemer, P. L., & Crawford, P. (2007). Student perceptions of active learning in a large cross-disciplinary classroom. Active Learning in Higher Education, 8(1), 9-30. https://doi.org/10.1177/1469787407074008

Majumdar, S. K., Rosenfeld, L. M., Rubba, P. A., Miller, E. W., & Schmalz, R. F. (Eds.). (1991). Science education in the United States: Issues, crises, and priorities (pp. 92-105). The Pennsylvania Academy of Science.

Messick, S. (1994). Foundations of validity: Meaning and consequences in psychological assessment. European Journal of Psychological Assessment, 10(1), 1-9.

Moore, R. W., & Foy, R. (1997). The scientific attitude inventory: A revision (SAI II). Journal of Research in Science Teaching, 34(4), 327-36. https://doi.org/10.1002/(SICI)1098-2736(199704)34:4<327::AID-TEA3>3.0.CO;2-T

Myers, R. E., & Fouts, J. T. (1992). A cluster analysis of high school science classroom environments and attitude toward science. Journal of Research in Science Teaching, 29(9), 929-937. https://doi.org/10.1002/tea.3660290904

National Academy of Sciences. (2007). Rising above the gathering storm: Energizing and employing America for a brighter future. The National Academies Press.

Neter, J., Wasserman, W., & Kutner, M. H. (1989). Applied linear regression models (2nd ed.). Irwin.

Newell, A.D., Zientek, L.R., Tharp, B.Z., Vogt, G.L, & Moreno, N.P. (2015). Students’ attitudes toward science as predictors of gains on student content knowledge: Benefits of an after-school program. School Science and Mathematics, 115(5), 216-225. https://doi.org/10.1111/ssm.12125

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.

Opp, K. (2019). The rationality of political protest: A comparative analysis of rational choice theory. Routledge.

Osborne, J., Simon, S., & Collin, S. (2003). Attitudes towards science: A review of the literature and its implications. International Journal of Science Education, 25(9), 1049-1079. https://doi.org/10.1080/0950069032000032199

Robbins, M. E., Schoenfisch, M. H., Moore, J. T., & Tolar, D. (2005). An interactive analytical chemistry summer camp for middle school girls. Journal of Chemical Education, 82(10), 1486-1488.

Schiefele, U. (1991). Interest, learning, and motivation. Educational Psychologist, 26(3 & 4), 299-323. https://doi.org/10.1207/s15326985ep2603&4_5

Schoor, C., & Bannert, M. (2011). Motivation in a computer-supported collaborative learning scenario and its impact on learning activities and knowledge acquisition. Learning and Instruction, 21(4), 560-573. https://doi.org/10.1016/j.learninstruc.2010.11.002

Scott, J. (2000). Rational choice theory. In G. Browning, A. Halcli, & F. Webster (Eds.), Understanding contemporary society: Theories of the present (pp. 126-138). Sage.

Simpson, R. D., Koballa, Jr., T. R., Oliver, J. S., Crawley, F. E. (1994). Research on the affective dimension of science learning. In D. Gabel (Ed.) Handbook of research on science teaching and learning (pp. 211–234). Macmillan.

Sherkat, W. E., & Wilson, J. (1995). Preferences, constraints, and choices in religious markets: An examination of religious switching and apostasy. Social Forces, 73(3), 993-1026. https://doi.org/10.2307/2580555

Swarat, S., Ortony, A., & Revelle, W. (2012). Activity matters: Understanding student interest in school science. Journal of Research in Science Teaching, 49(4), 515-537. https://doi.org/10.1002/tea.21010

Talisayon, V. M., Mulemwa, J., Corrigan, D., Mehta, J., Jenkins, E., Koulaidis, V., Goel, V., Aikenhead, G., & Ogawa, M. (2004). Relevance of science education. http://www.ils.uio.no/english/rose/about/rose-brief.html

Talton, E. L., & Simpson, R. D. (1986). Relationships of attitude towards self, family, and school with attitude toward science among adolescents. Science Education, 70(4), 365-374. https://doi.org/10.1002/sce.3730700403

Unfried, A., Faber, M., Stanhope, D. S., & Wiebe, E. (2015). The development and validation of a measure of student attitudes toward science, technology, engineering, and math (S-STEM). Journal of Psychoeducational Assessment, 33(7) 622-639. https://doi.org/10.1177/0734282915571160

Weinburgh, M. H., & Steele, D. (2000). The modified attitudes toward science inventory: Developing an instrument to be used with fifth grade urban students. Journal of Women and Minorities in Science and Engineering, 6, 87-98. https://doi.org/10.1615/JWomenMinorScienEng.v6.i1.50

Welch, A. G. (2010). Using the TOSRA to assess high school students’ attitudes toward science after competing in the FIRST robotics competition: An exploratory study. Eurasia Journal of Mathematics, Science & Technology Education, 6(3), 187-197. https://doi.org/10.12973/ejmste/75239

Wigfield, A. (1994). Expectancy-value theory of achievement motivation: A developmental perspective. Educational Psychology Review, 6(1), 49-78. https://doi.org/10.1007/BF02209024

Wittek, R. (2013). Rational choice theory. In Theory in social and cultural anthropology: An encyclopedia (pp. 686-689). Sage.

Downloads

Published

2021-09-29

Issue

Vol. 16 No. 4 (2021)

Section

Research & Evaluation Studies

License

Authors who publish with this journal agree to the following terms:

  1. The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
  2. Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
  3. The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons Attribution 4.0 International License or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
    1. Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
    with the understanding that the above condition can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
  4. The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
  5. Authors are permitted and encouraged to post online a prepublication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work. Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
  6. Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
  7. The Author represents and warrants that:
    1. the Work is the Author’s original work;
    2. the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
    3. the Work is not pending review or under consideration by another publisher;
    4. the Work has not previously been published;
    5. the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
    6. the Work contains no libel, invasion of privacy, or other unlawful matter.
  8. The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.

Revised 7/16/2018. Revision Description: Removed outdated link.