Examining 4-H Robotics in the Learning of Science, Engineering and Technology Topics and the Related Student Attitudes

Youths’ natural fascination and identification with robots make them an ideal teaching and learning platform. Robots would seem to be excellent hands-on tools to teach science, engineering and technology (SET) concepts. However, while research supports their use to increase interest and motivation, the effectiveness of robots to directly teach science, engineering, and technology concepts is less clear. The purpose of this study was to measure the effectiveness of a 4-H robotics program to support the learning of specific SET concepts and to examine related student attitudes towards science. This study compared the pretest and posttest scores on an assessment of basic SET concepts and attitudes of youth who participated in the 4-H robotics intervention with the scores of youth in a control group. Analysis of covariance (ANCOVA) results revealed that youth in the robotics intervention scored better on a SET concepts posttest. Student attitudes toward science were also investigated. The results suggested that educational robotics can engage youth in activities that support their learning of SET topics, but that it may have more limited impact on general student attitudes towards science, as measured by the study's attitudinal instrument.


Introduction
American students' low proficiency level in science, engineering, technology, and mathematics is a major impediment to the nation's 21st century global competitiveness.In the 2005 National Assessment of Educational Progress (NAEP) report, only two percent of American students attained advanced levels of science or mathematics achievement by Grade 12. Alarmingly, a substantial percentage of students also scored below the "basic" level of proficiency.In science, 32 percent of Grade 4 students, 41 percent of Grade 8 students, and 46 percent of Grade 12 students scored below the "basic" level.In mathematics, 20 percent of Grade 4 students, 31 percent of Grade 8 students, and 39 percent of Grade 12 students scored below this lowest level of proficiency.
When compared to other countries, American students routinely fall below the average levels of achievement (Lemke, Sen, Pahlke, Partelow, Miller, Williams, et al., 2004).America's economy is highly dependent upon technology.Unfortunately, the United States is producing fewer science and technology workers while other countries are steadily increasing the number of graduates in these fields (Porter & van Opstal, 2001).If technological and scientific innovation is to continue to drive the U.S. economy, there is a vital need for our educational system to engage in innovative practices that increase science, engineering, technology, and mathematics learning, and entice youth into these important career areas (Bonvillian, 2002).
We believe that one innovative way to help improve SET learning outcomes and to help youth form positive attitudes related to SET areas may be the use of robotics.Lower cost and design improvements in various robotics platforms, combined with widespread availability of robotic kits as learning tools make it possible for youth to explore and discover a wide range of exciting science, engineering, technology, and mathematics concepts.This study examined the potential of using robotics in 4-H after school and 4-H club programs to teach SET concepts and to examine changes in attitudes towards science.

Review of Literature
Much of the groundwork for using robotics in education was laid by Seymour Papert (1980).Using the Logo programming language and a robot called "Turtle" (because it was designed with a turtle-like plastic shell), Papert created an environment where children programmed computers and robots.According to Papert (1980) children gained a sense of power over the technology.
Research in the use of robotics in education has its roots in case studies that illustrate the potential effectiveness of robotics to impact both SET learning and motivation (Fagin, & Merkle, 2003).Past research reveals that robotics can generate a high degree of student interest and engagement and potentially promote interest in math and science careers (Nourbakhsh, Crowley, Wilkinson, & Hammer, 2004;Robinson, 2005;Rogers, & Portsmore, 2004).The robotics platform may also support a more personalized comprehension of scientific and mathematic principles through experimentation (Rogers, & Portsmore, 2004), encourage problem solving (Barnes, 2002;Mauch, 2001;Nourbakhsh, Crowley, Bhave, Hamner, Hsium, Perez-Bergquist, Richards, & Wilkinson, 2005;Robinson, 2005;Rogers & Portsmore, 2004), and promote cooperative learning (Nourbakhsh, et al., 2005;Beer, Chiel & Drushel,1999).Studies also underscore robotics' potential to engage females and underserved youth in SET learning, suggesting female students are more likely to appreciate learning with robots than traditional SET teaching techniques (Rogers, & Portsmore 2004;Nourbakhsh, et al., 2005).

Impact on Learning
Aside from rousing the imagination on a potential context for learning, what does the research really say about using robotics and its impact on SET learning?Unfortunately, much of the research available is lacking in the necessary rigor and depth to make solid conclusions.For example, Nourbakhsh et al. (2005) reported that a seven-week robotics course for high-school students led to significant increases in learning based on self-evaluations from the beginning of the course to the end.While the students may have felt they increased their knowledge, research also suggests that self-assessments have only a moderate relationship to performance (Dunning, Heath, & Suls, 2004;Mabe, & West, 1982).Gains reported in the study may not be an accurate reflection of the actual learning, as students may have over-estimated or underestimated their performance during the intervention.Fagin and Merkle (2003) examined the effect of robotics on teaching an introductory computer science course at the college level.The researchers compared computer programming laboratory midterm and final exam scores between groups that used the robotics, and groups that did not.The results indicated that laboratory sections using robots had significantly lower scores in the course than students who had not been taught with robots.However, the robotics sections were only allowed to practice problems in the laboratory, while students in the control sections had no such restriction.Students in robotic sections reported being frustrated by the lack of time to practice and felt that they were at a disadvantage compared to the other non-robotic laboratory sections.Given that the control and experimental groups were treated differently, the results of the study may not accurately reflect the impact robotics can have on learning.Barker and Ansorge (2007) examined the used of robotics in a 4-H after school program with middle school students.The results of their pilot study indicated a significant increase t(22.17)=12.93,P<.0001 in posttest scores as compared to a control group.However, the study used a low sample size n=14 and the results could not be generalized to the population of after school programs as a whole.

Attitudes
There is growing interest in examining students' attitudes towards learning, with recognition that affect surrounds cognition and can moderate learners' conceptual change (Also, & Watts, 2003;Koballa, & Glynn, 2007).Measuring a student's attitude is not a trivial matter; however, much of the robotics literature looking at attitudes relies heavily on subjective observation.For example, Rogers and Portsmore (2004) reported that using robotics as an outreach activity in elementary schools increased confidence and interest in mathematics and science.This conclusion was based on teacher perceptions; the researchers did not directly gather any data from individual participants.
In a similar study, Robinson (2005) interviewed three 8 th grade science teachers using robotics in a physics course.The interviews revealed that all three teachers believed students were more interested in science.However, no individual data was collected from the students to support the teachers' supposition.
Finally, Beer, Chiel and Drushel (1999) indicated that the success of their college robotics engineering course excited students about careers in engineering and science.Excitement, the researchers concluded, was based on an increased demand for the course and the offering of outreach activities to younger students.However, the researchers did not indicate if this observed excitement translated into increased interest in science.Moreover, the course and outreach activities that they described were geared towards robotic competitions between teams.It may be possible that students were primarily excited about the academic competitions, rather than the robotics or pursuing careers in science and engineering.
Although several instruments have been developed to assess scientific attitudes of youth, the Scientific Attitude Inventory (SAI I/II) is probably the most widely employed.Developed in 1970 by Moore and Sutman, the SAI (I) consisted of a 60 item, four-point Likert-type Scale with a series of 12 statements of attitude called "position statements."These twelve position statements assess six scientific attitudes -three based on intellectual attitudes and three based on emotional attitudes.The six attitudes include laws of science; scientific explanation; manner of scientific observation; value of scientific activities; usefulness of science to society and student career aspirations.Each scientific attitude has a positive and a negative scale to create the 12 position statements used as potentially measurable constructs.
Each position statement in the SAI contains several short "attitude statements" that query the student's attitudes towards science.The final 60 attitude statements were selected from an initial pool of 112 items submitted to a panel of judges and a group of high school students (Moore, & Sutman, 1970).Moore and Sutman (1970) state that they selected five attitude statements for each of the 12 position statements based on which items "received the greatest support from the judges and were not unanimously endorsed or rejected by the students."Moore and Sutman (1970) claimed content validity through the judges and students determination of the attitude statement's relationship with the position statement to which it is paired.Reliability was measured through the use of the Winer test-retest method using the pre and posttest scores of the control group resulting in a test-retest reliability coefficient of .934(Moore, & Sutman, 1970).
Moore and Foy revised and updated the SAI in 1997.This improved SAI (known as "SAI II") was been shortened to 40 items from the original 60 to make the instrument more usable.Verbiage was edited in several of the attitude statements to reduce "gender-biased language" and increase readability for ease of use (Moore, & Foy, 1997).They also incorporated several of Nagy's suggestions (1978) including changing the four-point Likert-type Scale to a five-point scale to account for the possible "undecided/neutral" response.Moore and Foy did not re-test the instrument for content validity, citing the statements used in the SAI II were "essentially the same as those used in the [original] SAI" and therefore the results received from the original instrument validation process are still valid (Moore, & Foy, 1997).
Clearly, the research in the use of robotics indicates there is a potential to increase both conceptual knowledge and attitudes towards science, engineering, and technology.However, there is a need to better quantify these effects in more controlled settings.Such research is needed to fully understand the learning impact of using robotics in both formal and non-formal environments.Studies that accurately and objectively provide insight into concept achievement, and the related impacts on attitudes and interest are needed to provide a better understanding of how to use this unique tool effectively.

The 4-H Robotics Program: An Overview
Nebraska State 4-H has worked with 4-H after school programs and 4-H clubs throughout the state to test the effectiveness of robotics in an informal educational environment.The robotics intervention is composed of a newly developed National 4-H Cooperative Curriculum System (CCS) robotics curriculum and a kit of robotic components from LEGO named LEGO Mindstorms.The LEGO Mindstorms kit consists of 828 parts including axles, gears, motors and sensors.The kit includes a programmable microcomputer with three output and three input ports for controlling sensors and motors.In addition, the robots are programmed using a specialized and modular programming language called ROBOLAB.The 4-H robotics curriculum contains 28 lessons designed around the Mindstorms kit.Beginning with simple building and programming challenges, it culminates with advanced robotic programming and engineering topics.

Purpose and Research Questions
This study examined the effects of an informal 4-H robotics intervention on the learning of selected topics in science, engineering and technology by students age 7-11 and on the student's science attitudes.Specifically, the following two research questions were: • What is the impact of the robotics instruction in 4-H after school programs and 4-H clubs in promoting learning in science, engineering, and technology (SET) for youth age 7-11?• What was the impact of the robotics instruction on students' attitudes towards science?

Methodology
Participants A total of 121 students from nine different schools represented the experimental group, while another 36 students acted as a control group.To provide some similarity in a comparison group, the 36 additional students were selected from three separate after school programs that did not participate in the robotics intervention.Students were drawn from three 4-H clubs and six after-school programs from four towns throughout Nebraska.Schools chosen were relatively similar in size (around 200 -300 students per school) and were well experienced in conducting 4-H related activities.Students ranged in age from 7 to 14 for the experimental group and from 11 to 14 for the control group.

Instrumentation
The instrumentation for the study was a 24-item, paper-and-pencil, multiple choice robotics assessment covering a variety of SET concepts covered in the curriculum.This assessment was developed by a team of 4-H facilitators, university professors, and two experts from Carnegie Mellon's University Robotics Academy.The instrument was refined in a pilot study of 32 elementary students, where a Cronbach's alpha reliability coefficient of .86 was demonstrated for the pilot instrument (Barker, & Ansorge, 2007).An overall Cronbach's alpha reliability coefficient of .667was demonstrated with the data retrieved for this study using that instrument.The lower alpha is attributable to low coefficients in two of the experimental sites.
When the two sites were dropped, the overall coefficient was calculated at .687.Furthermore, when test item 23 was removed, an alpha coefficient of .702was demonstrated.
As described previously, student attitudes were measured using the Scientific Attitudes Inventory (SAI II) (Moore, & Foy, 1997).This instrument taps into dimensions known as the nature of science, which considers ideas that are important to the development of a scientific literate citizenry (American Association for the Advancement of Science, 1990;Lederman, 1992).The instrument considers, for example, the possibility of scientific ideas changing over time, the idea that science cannot answer all questions, and the understanding that creativity is a part of scientific work.
The instrument, containing 40 Likert-type items and six underlying scales, has been widely used in science education research.The alpha reported by the inventory authors was .78; the alpha obtained from this research with a 4-H population was .80.

Robotics Intervention
The Nebraska 4-H robotics intervention is a statewide initiative available through county extension offices.4-H after school programs or clubs can request the robotic kits and laptop computers through the county educator for a small rental fee for eight weeks.State 4-H office staff provided training to program leaders with a four-hour, hands-on workshop where participants completed several lessons from the curriculum.Because each site was unique in its implementation of the program, the number of hours varied slightly from site to site.In general, the total robotics intervention, as delivered to the experimental group, represented 15 to 20 contact hours.
The study followed the accepted 4-H programmatic model (which focuses on experiential learning with opportunities to reflect and apply newly gained knowledge), and operated within an eight-week after school and 4-H club setting where students met at least once per week.
Students worked in teams of three to four in the first part of the study.The feedback from program leaders and students indicated that not all of the youth could participate with teams of this size, so some students also began working in teams of two as recommended by the program leaders.In all cases, the students followed step-by-step instructions for each lesson activity, which typically covered robot building and programming challenges.
Activities in the 4-H robotics program covered a variety of topics.For example, students explored the difference between robots, computers, and machines.Numerous computerprogramming concepts were introduced including firmware, pseudocode, loops and jumps, variables or modifiers, and multitasking.With each concept introduced, youth were encouraged to demonstrate knowledge acquisition by completing a specific robotic task.For example, youth learned that programming loops repeat sections of a program.To test this knowledge a square track was built and youth had to program their robots to complete eight laps around the four sections of the track.Using loops and modifiers, youth programmed their robots to move forward 36 inches and then execute a 90-degree right turn.Using a loop and a counter variable the section of code was repeated 36 times.The 4-H robotics program also reiterated mathematical concepts such as calculating averages, greater than and less than values, and calculating ratios.Finally, engineering-related concepts such as calibration and gears were also covered in the program.

Data Collection
The learning and attitude questionnaires were administered by after school teachers and 4-H staff members to each group prior to the beginning of the intervention (pretest).After eight weeks, the instruments were again administered to both groups (posttest).

Data Analysis Procedures
Separate analyses were conducted for each of the two research questions (student concept learning and student attitudes) related to the robotics intervention.The study used a pretestposttest quasi-experimental design for each of these two investigations, with the robotics test and attitude questionnaire acting as both a pretest and posttest in each of the experimental and control groups.The robotics test used a total score for the number of items correct; while scale and total scores were computed for the SAI.The primary analysis for each research question was a one-way analysis of covariance (ANCOVA) examining the differences in post measures between the experimental and control groups.The ANCOVA used the pre-measures as covariates, to adjust for initial differences between the two conditions.Student age was also included as a covariate and student gender as a fixed factor to help examine and consider those possible effects.Further examinations on the robotics test also included descriptive statistics for individual concepts reflected in the questions and the related item response tables.

Results
The results of the study supported that the robotics intervention, in this context of 4-H afterschool programs and 4-H clubs, is a promising approach for supporting SET-related learning as reflected on the robotics test.The ANCOVA analysis related to this student learning examined the effect of the robotics intervention using the posttest score of the concepts test as the dependent variable, and using the pretest score and age as covariates.Gender was also entered into the analysis as an additional fixed factor.The main effect for the study group was significant (F(1,141) = 11.04,p = .001),with the robotics intervention group scoring higher than the control group (m = 11.09,sd = 3.80; and m = 10.68,sd = 3.93 respectively).The pretest was a useful covariate (F(1,141) = 47.22,p = .001),with an initial score advantage suggested for the smaller control group of approximately two questions.Age was shown to not be a significant covariate (F(1,141) = 2.31, p = .13)and gender was not a significant factor (F(1,141) = 8.33, p = .478).A breakdown of student responses related to particular SET concepts (as represented by the individual question summaries) was also conducted.The results suggested that the experimental group did better on questions that included programming concepts (such as looping, documentation and multi-tasking), several mathematics and science concepts (including fractions and ratios), and the engineering topic of gears.On a few questions the control group scored higher than the experimental group.The higher scores are most likely the result of random response patterns.
In contrast to the learning results, the total attitudinal analysis score showed no significant difference between treatment (m = 142.67,sd 15.03) and control (m = 137.22,sd = 12.34) conditions F (1,75) 1 = .55,p = .46 .This lack of significance held for both the total attitudinal score, as well as each of the six underlying scales.When compared to the control group the treatment group did not report any significant increases in intellectual attitudes or emotional attitudes towards science.Clearly participation in the after-school and 4-H robotics program did not impact attitudes towards science, as measured by the SAI, with this population of middle school students.

Table 2
ANCOVA Analysis of Concepts Posttest (as Dependent Variable)

Table 3
ANCOVA Analysis of Attitudes Post Survey (as Dependent Variable) R Squared = .398(Adjusted R Squared = .381)