Science Comprehension Retention Among Youth Agriscience Students Instructed in Weather and Climate

The purpose of this article is to examine the science comprehension retention of 8-grade science students taught a new weather and climate curriculum. The students’ middle school is part of an innovative Extension youth agricultural science center that has a mission to develop and test new teaching and learning models and curricula in agriculture and natural resources. Our curriculum was developed following a science comprehension model we created and have been testing at the center. It contained lessons on the water cycle, the greenhouse effect, measuring and analyzing precipitation and temperature data, and mitigating and adapting to weather and climate extremes in agriculture and natural resources. For each lesson, students viewed introductory PowerPoint slides, participated in an activating strategy, set up an experiment or analyzed local precipitation or temperature data, formulated hypotheses, participated in a summary activity, and completed a worksheet. We pretested 81 students, Journal of Youth Development | http://jyd.pitt.edu/ | Vol. 15 Issue 6 DOI 10.5195/jyd.2020.902 Weather & Climate Science Comprehension Retention 117 taught the curriculum over a 6-day period, and gave the posttest. We returned 2 months later to administer a follow-up to check for science comprehension retention. The students’ overall science comprehension and science knowledge, science skills, and reasoning abilities subcomponent follow-up scores were lower than their post-program test scores. Both boys and girls declined in their overall postprogram test gains over the 2 months. Students also declined in their preference for learning-by-doing from post-test to follow-up. Based on these results, we made changes to the curriculum consistent with the literature on learning retention before publishing it online for youth educators.

developing, testing, and diffusing an experiential and inquiry-based weather and climate science curriculum for middle school-aged youth as part of the overall climate science efforts between NMSU and the Southwest Climate Hub, prompted us to develop the curriculum as our 2014-2019 Hatch/Agricultural Experiment Station project.
We utilized the NMSU Extension and Research Youth Agricultural Science Center (YASC) in Las Vegas, New Mexico (Skelton & Dormody, 2009), to develop and pilot test the weather and climate science curriculum. The center is a partnership between the New Mexico Cooperative Extension Service, Las Vegas City Schools, and the State of New Mexico with the following purpose: The YASC is a youth science center emphasizing inquiry-based learning and experiential education. A basic premise of the mission is to develop a teaching and learning model of excellence for agriculture and natural resource science that complements in-class instruction by providing context to content through hands-on learning opportunities. The YASC is engaged in earth, life and physical science teaching and learning. STEM-based education is delivered through teaching the principles and applications of sustainable agriculture, renewable energy, and local food systems (New Mexico State University, n.d.).
In keeping with the center's purpose, we use it to develop and test teaching approaches, learning models, and curricula with Las Vegas City Schools students and teachers. For example, we conceptualized and are testing a teaching and learning model at the center to improve science comprehension (National Research Council, 2000;Skelton et al., 2012). We started by defining science comprehension as the interaction between science knowledge, science skills, and scientific reasoning abilities. The basic premise of the center's science comprehension model ( Figure 1) is that when students engage in an (a) inquiry-based and (b) experiential lesson designed to improve (c) science knowledge, (d) science skills, and (e) science reasoning abilities in a content area, their science comprehension will increase (Skelton et al., 2012). With each additional lesson developed and taught by integrating the model's five supporting elements listed above, science comprehension should also increase over time, emulating in three dimensions a rising and expanding "learning tornado" (Skelton, Dormody, & Dappen, 2016, p. 85). Hence, the weather and climate lessons developed and pilot-tested in this study all integrate the five supporting elements of the science comprehension model.

Figure 1. YASC Conceptual Model for Improving Youth Science Comprehension
We conceptualized the model from various theories on experiential learning (Bourdeau, 2004;Cervetti et al., 2006;Kolb, 1984;McLeod, 2017;Swinehart, 1992) and inquiry-based learning (National Research Council, 2000;Pedaste et al., 2015;Wilhelm & Beishuizen, 2003). In experiential learning, students undergo a four-stage learning cycle (i.e., concrete experience, reflective observation, abstract conceptualization, and active experimentation) that creates new understandings (Kolb, 1984;McLeod, 2017). While in an experiential learning cycle, students have and reflect upon an experience, and make modifications to and test their new understandings. Inquiry-based learning is a strategy to teach students to inquire and think in a manner like scientists. "In this process, students often carry out a self-directed, partly inductive and partly deductive learning process by doing experiments to investigate the relations for at least one set of dependent and independent variables" (Wilhelm & Beishuizen, 2003, p. 382). The Figure 1 model was previously tested at the center by Skelton, Dormody, and Lewis (2016); andDormody et al. (2020a, 2020b). All four studies indicated positive effects on science comprehension when designing and teaching curricula following the model. Dormody et al. (2020b) found that overall science comprehension scores; science knowledge, science skills, and reasoning abilities subcomponent scores; and scores on four of the five lessons improved from pretest to posttest when a model-based five-lesson weather and climate curriculum was taught to a sample of mostly Hispanic and economically disadvantaged eighth-grade science students. Dormody et al. (2020a) found that scores for the five lesson worksheets in their weather and climate curriculum were related to overall science comprehension improvement scores, science knowledge improvement scores, and improvement scores related to the lesson on mitigating and adapting to weather and climate extremes in agriculture and natural resources. A limitation to these studies was their use of case study and one-group pretest-posttest designs without a control group. These designs cannot be used to determine causality (Campbell & Stanley, 1963).
The present study is an extension of the Dormody et al. (2020aDormody et al. ( , 2020b studies and focused on science comprehension retention after students were taught the weather and climate curriculum. A third administration of the test (follow-up) 2 months after delivering the treatment and posttest would give us useful information for strengthening the curriculum. Differences between male and female students on science comprehension retention could also be determined.
Gender differences in youth science achievement persist in public education. In a national study, Hyde and Lynn (2006) found that from Grades 4 through 12, boys performed consistently better than girls on the science portion of the National Assessment of Educational Progress (NAEP), although the effect sizes by grade level were small. In another national study of middle school earth, physical, and life science achievement using National Center for Educational Statistics ECLS-K:99 data, Quinn and Cooc (2015) found science achievement gaps by gender that were small and stable from elementary through middle school grades, narrowing slightly by the eighth grade. They stated, "Our findings indicate that the 'leaky' science pipeline may begin as early as third grade, suggesting that interventions aimed at closing gaps should begin when students are young" (p. 334). Lee and Burkham (1996) recommended weekly laboratory experiences as a way to narrow science achievement gender gaps. "Such laboratory experiences are especially beneficial for girls' achievement in physical science, but not boys'" (p. 613). Consistent with their recommendation, we incorporated three experiments and two long-term climate data analyses into our five weather and climate science lessons.
Recent educational research has also focused on gender gaps for other variables related to science achievement. After participating in a summer STEM program for average and underperforming middle school students, significant gaps between girls and boys were eliminated on 12 of 15 STEM interest indicators. The three remaining indicators with significantly lower female responses were (a) "enjoying doing things in science," (b) "enjoying learning about science," and (c) "liking learning about atoms and molecules" (Naizer et al., 2014, p. 31). For high school-aged youth in the Philadelphia Adolescent Life Study, girls reported greater science task value than boys while the boys reported higher self-concept and expectations than girls for success in both math and science (Else-Quest et al., 2013). Britner (2008 found that for both high school boys and girls, science self-efficacy was a significant predictor of science course grades. In a study of 1,300 southwest Minnesota middle and high school students, Desy et al. (2011) found that middle school girls had more anxiety toward science and were more likely to indicate science as a least favorite subject than boys. However, about 40% of the girls indicated a top career interest in a STEM field or teaching compared to about 30% of the boys. Gender gaps in youth science achievement and related variables remain an area in need of further research and development that we wanted to address in this study.
Finally, the test used in the present study and Dormody et al. (2020b) also contained a question on students' learning mode preference to determine if our inquiry-based and experiential lessons had a relationship with learning mode preference. Dormody et al. (2020b) determined the number of youths with a preference for learning by doing increased from pretest to posttest. The present study followed up to determine students' learning mode preferences 2 months after the treatment and posttest.

Purpose and Research Objectives
The purpose of this study is to determine eighth-grade students' weather and climate science comprehension retention levels 2 months after being taught our curriculum and to use the results to strengthen the curriculum before making it available to youth educators. The primary objectives were to determine whether several indicators of science comprehension changed between the posttest (immediately following delivery of the curriculum) and follow-up (2 months later), including • overall science comprehension • comprehension on three subcomponents, namely science knowledge, science skills, and reasoning abilities • comprehension on five specific lessons: the water cycle (Lesson 1), the greenhouse effect (Lesson 2), measuring and analyzing precipitation and temperature (Lessons 3 and 4), and mitigating and adapting to weather and climate extremes in agriculture and natural resources (Lesson 5) Secondary objectives were to test • gender differences in science comprehension from pretest to posttest and from posttest to follow-up • changes in learning mode preference from posttest to follow-up

Method Research Design and Participants
We employed the one-group pretest-posttest design (Campbell & Stanley, 1963) in the pilot test of the weather and climate science curriculum. The design would provide the information needed to strengthen the curriculum, while meeting our budget and time constraints. From an equity perspective, the design gave all participants the benefit of experiencing the curriculum.
Although commonly used for testing curricula, the design has the limitation of not using a control group. Hence, it is susceptible to internal validity threats and cannot be used to determine causality. To strengthen the study on internal validity, we implemented various control measures. Immediate administration of the treatment after the pretest followed by immediate administration of the posttest helped control history and maturation threats. Testing threat was controlled by making the treatment robust (e.g., six active 50-minute class periods of teaching after the pretest). The same test was used and administered the same way as pretest, posttest, and follow-up to control instrumentation threat. The most problematic threats to internal validity in the present study were history and maturation during the 2 months from posttest to follow-up.
For the pilot, the target population was 120 eighth-grade students grouped into five science classes at Memorial Middle School in Las Vegas, New Mexico. Eighty-one (67.50%) submitted informed assent and consent forms and completed the pretest, posttest, and follow-up. All 81 were classified as economically disadvantaged. Seventy-two (88.89%) were Hispanic; six (7.41%) were Caucasian; and one (1.23%) was either Asian, African American, or Native American. Thirty-four (41.98%) were female and 47 (58.02%) were male. Twelve of the students (14.81%) were categorized as special needs with one of these students listed as gifted.

Treatment
After students took the pretest, the five-lesson curriculum was taught by one of the researchers to the five classes over six 50-minute class periods per class. To ensure that the curriculum was research-based, content was developed from government agency websites and research reports, and from research-based material taught by the New Mexico climatologist in his university course and outreach education programs on weather and climate (Dormody & Skelton, 2019). Lessons were aligned with matching middle school Next Generation Science Standards (n.d.) and Agriculture, Food, and Natural Resources Content Standards and Benchmarks for New Mexico (Castillo, 2003). Each lesson integrated the five components of the Figure 1 science comprehension model (Skelton et al., 2012). All lessons featured a few introductory PowerPoint slides shown on a Smart TV, an activating strategy, setting up an experiment or conducting local weather and climate data analyses, worksheets, and a summary activity.
Lesson 1 was on the water cycle. It featured a water cycle role play and an experiment to depict the effects of flooding, drought, and typical precipitation events on corn growth.
Students formulated hypotheses after setting up the experiment. Lesson 2 was on the greenhouse effect. It featured a greenhouse effect role play followed by setting up a greenhouse effect bean growth experiment and formulating hypotheses. Lessons 3 and 4 were on weather stations and how to access and analyze online local precipitation and temperature data. The students were taught to navigate the National Oceanic and Atmospheric Administration (NOAA) Regional Climate Centers database (National Oceanic and Atmospheric Administration, n.d.) using I-Pads and find local weather station precipitation and temperature data for single days and 1 year, and to develop and test hypotheses for 70 years of precipitation and temperature in their local area. Lesson 5 was on how to mitigate and adapt to weather and climate extremes in agriculture and natural resources. It featured examples of adaptation and mitigation strategies, learning to use the handheld infrared thermometer to measure surface temperatures, and setting up an experiment to test temperatures and moisture levels of potting soil covered by different colored garden mulches placed under fluorescent lights and heat lamps. Students again formulated hypotheses after setting up the experiment. Because scientists often work in teams on a research problem and the positive effects of collaborative or team-based learning on learning retention (Gonzalez, 2018;Lin, 2006;Morpus, 2017;Terada, 2017), we put the students in teams of four to set up the three experiments (Lessons 1, 2, and 5), and teams of two to share an I-Pad and complete their precipitation and temperature protocols (Lessons 3 and 4). The fifth lesson was followed by a 10-minute unit summary and then the posttest. Two months after the posttest, the follow-up was given in all five classes.
In preparation for the present study, we pilot-tested the three experiments in earlier years of the project with students that were not part of the present study. The intent was to ensure the experiments addressed the subcomponents of science comprehension (Skelton et al., 2012), could be set up during a class period, and yield measurable differences between experimental treatments and control Skelton et al., 2017). The temperature and precipitation data analysis activities using the NOAA Regional Climate Center database were also pilot-tested in an earlier year of the project to ensure they addressed the subcomponents of science comprehension and could be completed in a class period .

Instrumentation
The pretest/posttest/follow-up were made up of two multiple choice questions covering each of the following: science knowledge (the content taught in each lesson), science skills (related to the scientific skills taught in each lesson), and reasoning abilities (related to the hypothesis writing and testing completed in each lesson) for each of the five lessons. This yielded a total of subcomponent). To ensure content and face validity, the questions were drawn from the lesson objectives, and written to be consistent with the PowerPoint slides, worksheets, and activities developed for each lesson. The test was broken into two similar 15-question halves for splithalves reliability testing. Applying the Spearman-Brown Prophecy Formula to pretest, post-test, and follow-up data (n = 81) yielded split-halves reliability coefficients of .68 for the pretest, .72 for the post-test, and .81 for the follow-up.
Sample science knowledge, science skills, and reasoning abilities questions from the tests were, respectively: On average, how much has the human influence on the greenhouse effect increased the earth's temperature? 1. It has increased greatly (over 2 degrees F) 2. It has decreased greatly (over 2 degrees F) 3. It has increased slightly (between 0 and 2 degrees F)

Data Analysis
Overall science comprehension retention scores (Objective 1); science knowledge, science skills, and reasoning abilities retention scores (Objective 2); and lesson retention scores (Objective 3) were analyzed separately using a mixed model with a fixed effect for the test occasion (posttest scores by follow-up scores). The mixed model included random effects for class, class by test occasion, and student within class. Effect sizes were computed as the estimated posttest/followup difference divided by the model-based estimate of the difference standard deviation (Table   1).
For the Objective 4 analysis; pretest, posttest, and follow-up occasions were used and fitted to a mixed model with fixed effects for test, gender, and their interaction; and random effects for class, class by test, class by gender, and class by gender by test (Table 2). This model accounted for the lower variance at the pretest occasion by fitting an unstructured covariance to the repeated measures from students within classes. Change in students' preferred way to learn (Objective 5) was descriptively summarized using a cross-tab procedure with the proportion of students that changed their preference from posttest to follow-up estimated with a 95% confidence interval (Table 3). To do this, we created a dichotomous variable: yes (changed preference), no (did not change preference) and estimated the percentage of students that changed their preference. All objectives were analyzed using SAS version 9.4 software and significance was defined as p ≤ .05.

Objectives 1, 2, and 3
For the overall test (Objective 1), students declined an average of 2.07 points on the 30 items from the posttest to follow-up 2 months after being taught the curriculum (Table 1). The difference was significant and, following Cohen's (1988) (Table 1). For Lesson 4 on measuring and analyzing temperature, students declined an average of 0.49 points on the six questions from posttest to follow-up. The difference was significant and yielded an effect size of -0.42 (small to medium).
For the six questions on Lesson 5, mitigating and adapting to weather and climate extremes in agriculture and natural resources, students declined an average of 0.58 points from posttest to follow-up. The difference was significant and produced an effect size of -0.37 (small to medium).

Objective 4
For this analysis, test and the gender by test interaction effects were significant (F 2,8 = 39.12, p < .0001 and F 2,8 = 9.11, p = .0086, respectively) even though, averaged across test occasions, genders did not differ (F 1,4 = 0.86, p = .4073). While both females and males improved significantly from pretest to posttest, females improved by an average of 5.18 points (1.56 or very large effect size) while males improved by an average of 1.73 points (0.52 or medium effect size) on the 30-item test ( Table 2). The female improvement from pretest to posttest was 3.45 points (p = 0.0027) greater than for male students. higher on the follow-up than on the pretest while male follow-up scores were not detectably higher than their pretest scores (p = .7895).

Objective 5
Of the 77 students that answered this question on both the posttest and follow-up, 27.27% (n = 21) changed their learning mode preference on the follow-up (Table 3). A 95% confidence interval for the percent that indicated changing their preferred learning mode estimates that the percentage changing their mind would be between 17.3% to 37.2%, so this change in preferred learning mode between post-test and follow-up was significant. Table 3  follow-up, preferences for learning by reading, learning by observing, and learning by lecturing increased by three, four, and three students, respectively.

Discussion
Two months after being taught the weather and climate curriculum, overall science comprehension follow-up scores had declined from posttest scores for our predominantly Hispanic eighth-grade students. They declined for questions associated with the science knowledge, science skills, and reasoning abilities subcomponents of our science comprehension model (Skelton et al., 2012). There were no changes for questions associated with Lesson 1 (water cycle), Lesson 2 (greenhouse effect), and Lesson 3 (measuring and analyzing temperature), and declines for the questions associated with Lesson 4 (measuring and analyzing temperature, and Lesson 5 (mitigating and adapting to weather and climate extremes in agriculture and natural resources). Science comprehension scores increased for female and male students from pretest to posttest with a larger increase experienced by female students.
Female students retained about half of their posttest gains and male students returned close to pretest science comprehension levels on the follow-up.
A limitation to the fidelity of this study is that we were able to work with students only to set up and write hypothesis statements for the water cycle, greenhouse effect, and mitigation and adaptation lesson experiments. We were unable to oversee data collection and analysis activities. This was due to budget and logistical constraints that included limited travel funds; one researcher having to return to the university to teach; another researcher being reassigned away from Memorial Middle School; and the science teacher, who volunteered her five classes for 8 days of pilot test instruction and test administration, needing to take the classes back to cover required content. We estimate that data collection and analysis would have added approximately 90 more minutes of instruction after the posttest to the 300 minutes of instruction the students received before the post-test. That student scores on the follow-up declined for only one of the three lessons that had experiments, softens this limitation.
However, for Lesson 5 (mitigating and adapting to weather and climate extremes in agriculture and natural resources), subcomponent, and overall science comprehension scores, these additional minutes of instruction to collect and analyze experimental data, could have reinforced learning and potentially improved science comprehension retention.
A second limitation to the study is that the one-group, pretest-posttest design (Campbell & Stanley, 1963), lacks a control group and cannot be used to determine causality. We determined that the one-group, pretest-posttest design would provide the information needed to strengthen the curriculum before making it available to educators, while accommodating our budget and time constraints. We also selected the design for equity reasons. Because we were working with an economically disadvantaged population of youths that is traditionally underrepresented in STEM, we wanted all participants to receive the benefits of instruction over the weather and climate curriculum. To strengthen the study on internal validity, we implemented various control measures highlighted in the Research Design and Participants section. We could not control for history and maturation threats during the 2 months from posttest to follow-up. Taking into account these potential threats, the declines in science comprehension from posttest to follow-up found in this study were dramatic and useful in directing our thinking on how to improve the curriculum to achieve higher science comprehension retention.
Based on the science comprehension retention results from this study, we recommend that youth educators who teach our weather and climate curriculum adopt three strategies to improve science comprehension retention. The first is for educators to complete data collection and analysis activities for the three experiments as discussed above. The constraints the researchers had to overseeing data collection and analysis activities were specific to the study and should not burden youth educators in their classrooms or programs. The second is for educators to take more time on the reflective summary part of each lesson and go back over the students' worksheets and review every part of the lesson. We had approximately 50 minutes to teach each lesson and found that the lessons would benefit from additional time for reflective summary. After reflecting on the declines in science comprehension from posttest to follow-up, we added additional time for review and reflection to each lesson plan. Thirdly, we recommend that educators continue to refer back to the material covered in the curriculum with their students after the unit test. As other lessons are taught, educators should look for opportunities to connect students' prior learning on weather and climate science to other subjects. These three recommendations should make the lessons stronger in experiential (Morpus, 2017), inquiry-based (Johnson et al., 2012), problem-solving (Cheng et al., 2018), and cooperative or team-based learning (Gonzalez, 2018;Lin, 2006;Morpus, 2017;Terada, 2017) and more thoroughly use the formative worksheets the students completed (McDaniel et al., 2011) and writing to learn strategies (Gonzalez, 2018) that are reported in our literature review to enhance learning retention.
These recommendations should also work for female and male students. That the female students in this study had higher science comprehension than the male students on the posttest and follow-up is encouraging in closing the science achievement gender gap. From our literature review, a possible explanation that would require additional research to ascertain, is that we incorporated a major experiential and inquiry-based laboratory activity in each of the five lessons. Lee and Burkham (1996) found that for girls, laboratory experiences were particularly effective in improving physical science achievement. Further research would also be needed to determine if other parts of the lessons, like the worksheets or activating strategies, helped female or male students improve and retain science comprehension.
A significant number of students changed their learning mode preference to learning by doing from pretest to posttest when taught our highly experiential and inquiry-based lessons during the study (Dormody et al., 2020b). Two months later, the number of students who preferred learning by doing declined on the follow-up while preferences for learning by reading, observing, and lecturing all increased slightly. A possible explanation for this result could be that learning-mode preference is unstable in some youth and subject to the modes being emphasized by their teachers. Our recommendation to youth agriscience and science educators is to consistently use experiential and inquiry-based teaching approaches to develop science comprehension.
The purpose of this study was to determine eighth-grade students' weather and climate science comprehension retention levels 2 months after being taught our curriculum and to use the results to strengthen the curriculum before making it available to youth educators. We followed the recommendations from the study and two previous studies (Dormody et al., 2020a(Dormody et al., , 2020b to make final changes to the curriculum (lesson plans, worksheets, PowerPoint slides, unit test, and answer keys) which we then published on a new website (Dormody & Skelton, 2019). The curriculum can be used in formal education classrooms, county educator-led school enrichment programs, after-school and summer special interest programs, and with homeschooled youth.
Additionally, we are exploring developing the curriculum into a 4-H project format. We recommend further research on science comprehension when regular classroom teachers use the weather and climate curriculum. Additional research is also needed when our model (Skelton et al., 2012) is used to develop and teach other science content, and to explore the potential for three-dimensional growth in science comprehension  using the model.