Assessing the Civic Attitudes and Participation of Urban Elementary Students

Presented is a report of a study conducted to examine Common Cents’ Penny Harvest program in New York City public elementary schools. Penny Harvest is a service learning program designed to promote positive social and civic values among youth. The goal of this paper is to analyze the strengths and weaknesses of the research design in an effort to contribute new insight into effective and appropriate ways to measure civic-service-program success. Additionally, this work provides program results for the program evaluated. Our findings indicate that students in New York City public schools are highly involved in service projects – both in-school and outside of school. We present additional evidence on how such participation is related to a host of social and civic attitudes. Finally, we critique the research design used in this study and offer improvements to be made in future studies.


Introduction
The evaluation of service-learning and civic instruction has grown both in number and quality, over the last decade, with attempts to better understand and describe civic instruction and community service effects on student outcomes.For the most part, evaluations have found that service learning programs enhance a host of student outcomes, including students' sense of social responsibility, civic engagement, personal efficacy, academic learning, and critical thinking skills (Astin & Sax, 1998;Astin, Sax, & Avalos, 1999;Markus, Howard, & King, 1993).
Research has indicated that participation in a service learning program leads to a variety of personal development outcomes such as re-education of risk taking behaviors, self-efficacy, potency (one's belief that he/she can make a difference), resilience, social competence, acceptance of diversity, and related constructs (S.Billig, 2000).This work also has shown that youth who participate in such programs have enhanced social skills and more favorable attitudes toward adults (Hamilton & Fenzel, 1988).Similarly, Billig (2000) finds that service learning has positive effects in three domains: academic/cognitive, civic, and personal/social.
In general, service learning has positive effects for students in a variety of arenas (Astin & Sax, 1998;Scales, Roehlkepartain et al., 2006;Tannenbaum & Brown-Welty, 2006).For example, in one recent study (Scales, Blyth et al., 2000) differences between the participants and nonparticipants were found on measures of concern for others' welfare, positive perceptions of the opportunities for personal development in schools, pursuit of higher grades, and self-efficacy.Service learning was associated with higher gains in each of these areas over the period that the surveys covered.
Taken together, these studies provide evidence of the potential benefits of service programs for students.Results indicate that participation in such programs helped students develop their personal leadership skills, define their career goals, gain a better appreciation for their academic work, and develop a spirit of involving themselves in the community.Yet much work remains to learn about how these programs influence students and outcomes.
Despite this encouraging set of findings, the research designs employed to study the effects of service learning have significant limitations (S.Billig, 2002).Some of these limitations are related to the characteristics of the sample studied, particularly to sample size and sample selection.Additionally the ways service learning and its benefits are measured have been identified as restrictive (Alt, 1997;Koliba, Campbell, & Shapiro, 2006).One of the largest concerns with regard to samples is the likelihood of selection bias as most students and schools involved in such programs have self-selected into them and into evaluations of the programs.Additionally, there are often incomplete, as well as poorly constructed, data due to the nature of the population studied.Previous service learning evaluations have focused primarily, if not exclusively, on students in older grades (from grade 7 upward).This focus on older ages is the result of both the fact that most service learning programs are oriented toward older students and that the few programs focusing on younger grades have received little research attention.Only a handful of studies have looked at these issues for an elementary grade population (Chi Jastrzab, & Melchior, 2006).
One recent exploration into younger children's perception of service learning found that even among kindergarten children there were positive results associated with service learning activities (Smith, 2007).Similarly, the Michigan Learn and Serve study reports the impacts of service learning on students in grades 2-5, finding the effects for this age are stronger than for students in grades 7-12.This study finds that those involved were more engaged in English language arts classes than their counterparts not involved in such programs (S.Billig, 2004;S. H. Billig & Klute, 2003).In a study based in California, Furco (2002) documented significant differences between students who performed service and those who did not for each of the six educational domains measured by the survey (academic, career, civic, ethical, personal, and social).Students who performed service developed more positive attitudes toward school, themselves, others, the future, and their communities.
In this paper, we report on the results of a study of a service-learning program based in New York City's public schools: the Penny Harvest program.We first describe the model of the program, then provide and account of the innovative strategy used to research the program.We then report the findings from our analysis and conclude by analyzing the strengths and weaknesses of the research design.

The Penny Harvest Program
Penny Harvest is a multi-faceted program designed to transform the multi-million-dollar resource of idle pennies into the philanthropic property of children.By teaching students to recycle their community's pennies and use them to make a difference, the program is structured to develop a new generation of active citizens committed to social justice, tolerance, and democracy.The goals of the Penny Harvest program are broad, covering a wide range of possible outcomes.Students who participate are expected to improve in a variety of realms of personal beliefs: gaining self-confidence and self-awareness, increasing their sense of caring and social justice; feeling empowered to help ameliorate social problems, and reducing negative stereotypes of others different than them.In addition, students are expected to develop leadership ability and view themselves as more competent social actors in solving social problems.
Common Cents was founded in 1991 to promote the Penny Harvest, then an annual youth campaign to help homeless people by collecting pennies and using the funds to purchase food and clothing.Over the last 17 years, this community service evolved into the Penny Harvest Program -a versatile, year-long progression of classroom and school-wide service-learning experiences.A fully matured Penny Harvest Program engages very large numbers of students within a given community in resource mobilization, youth philanthropy, youth service, civic engagement and peer mentoring.

The Study
To examine the social, civic, and academic engagement of students participating in the Penny Harvest, we conducted a research study to interview students in grades 3 through 5 at a set of public schools across New York City.We developed a survey instrument that would facilitate comparisons between students in New York City and other locations.Students were interviewed twice during the school year: once in the fall semester, before the Penny Harvest program for the school year had begun, and once in the spring, after the program had finished for the year.

Sample of Schools and Students
We selected a set of schools based on a set of criteria, including geographic location (borough), characteristics of the student population, and features of the program's operation in the school.A group of ten well-established schools from the subset of all city elementary schools in which the program has been operating for more than two years were selected."Mature" schools were identified as we wanted to make certain that there was relative consistency in the program elements in the evaluation period and we wanted to reduce the possibility that our findings might be influenced by fidelity of implementation.
Because schools chose whether or not to participate in the Penny Harvest program, it was difficult to find appropriate comparison or control schools within the city.That is, the factors that are related to whether or not a school participates in the program, arguably, are likely to be related to the outcomes the program.Therefore, given the lack of suitable comparison schools within the district, we chose to focus only on schools with programs in our analysis.For comparison with non-Penny Harvest schools, we draw upon research instruments used in other districts, as described below.
Within each school, we sought to interview two classes per grade, administering the interview to all students in the class for whom parental permission to participate in the study was granted.The percentage of students in third through fifth grade that this selection process yielded varied substantially by the size of the school.For the smallest of the schools in the study, two classes represented the entirety of the grade, while for others, two classes was onefourth or less of the student population.

Research Instruments
We drew our questions from a set of surveys previously conducted in other urban settings.Most items were taken from the University of California at Berkeley's Service-Learning Research and Development Center's Civic Responsibility Survey National Survey of Student Engagement (Furco, Muller, & Ammon, 1998) and the MacArthur School Engagement Survey from Phyllis Blumenfeld and her colleagues at the University of Michigan (Blumenfeld, 1998).These two research instruments have been used on a variety of students in different settings.Moreover, as noted earlier, by using these instruments, we are able to compare students in Penny Harvest schools with students in other, similar districts.

Interview Protocol
Students participating in the study were asked to respond to a series of interview questions that evaluated attitudes and behaviors hypothesize to be related to participation in the Penny Harvest Program.In order to ensure that students understood each question and that linguistic and literacy differences did not influence the results, the interviews were administered in a group setting.A copy of the research instrument was distributed to each student so that their answers could be recorded privately.However, an interviewer read each question aloud and waited for students to complete their answers, a procedure that helped alleviate confusion about meaning.

Analysis
The analysis consisted of three parts: first, an examination of the results from data collected in the fall; second, a similar examination using data collected in the spring; third, a comparison of fall and spring, looking at changes and differences between schools and grades.For all analyses, we use fixed effects multiple regression, to control for the confounding influences of individual-level factors as well as potentially important between-school differences.

Results -Fall 2005
The first question addressed asked how many students participated in service learning activities prior to the 2005-06 school year.Although the Penny Harvest program is designed to increase student participation, we needed to know something about the number of students who have participated in service learning activities earlier for the sake of comparison.To gauge this, the survey instrument included two questions: one that assesses whether the student had participated in service projects in school in earlier years and another that asked whether the student had participated in service projects outside of school.
When examining the data for the fall 2005 wave, several factors stood out as key contributors to whether a student participates in service.These prominent factors were school, gender, and grade in school.Certain schools showed a greater likelihood of participating in service than others, and girls were more likely to have done a service project than their male counterparts.We used the chi-square test for independence to analyze the relationship between service performed outside of school and the variables for school (i.e., each individual school in the survey sample), gender, and grade (3rd, 4th, or 5th).School and gender both showed a significant correlation to service outside of school.School #4 demonstrated the highest level of service participation outside of school at 57.1% (n=21).The second highest level occurred at School #5, where 55.4% of students indicated taking part in service outside of school (n=74).School #3 showed the least amount of participation outside of school at 27.7% (n=141).Approximately 45.9% of girls said they did service outside of school (n=355) as compared to 37.9% of boys (n=314).Finally, the chi-square analysis showed that there was no significant difference between the participation in service outside of school and grade.That is, being in third, fourth, or fifth grade did not affect whether a student participated in service outside of school.Our sample was composed of 225 third graders, 225 fourth graders, and 220 fifth graders.
In addition to analyzing service outside of school, we examined service projects students completed in connection with the school the previous academic year.We utilized the chi-square test for independence to examine the relationship between school, gender, and grade and student participation in service in school in the last school year (service last year).All three comparison variables demonstrated a significant relationship with service in the previous year (2004-05).School #5 had a 97.3% level of participation in service during 2004-05 (n=74), the highest percentage in the sample.The second highest level of service before Penny Harvest's entrance occurred in School #9 at 93.0% (n=114).School #7 showed the lowest percentage of service previous to 2005 at 69.6% (n=102).The fact that School #7 reported the lowest amount of service before the 2005 school year, but still had nearly 70% participation illustrates that our student sample population came into the study with a prior history of taking part in service projects with their schools.
As was the case in service outside of school, girls in the sample participated more in service projects in 2004-05 than boys did.85.6% of girls (n=355) said they did service "last year", while 76.4% of boys (n= 314) reported doing so.Contrary to the lack of difference shown when examining grade and service outside of school, grade did impact service in the year before the interview.Fifth graders were the most likely to have done service before the program start at 90.5% (n=220).80.9% of third graders (n=225) said they did service "last year," while 72.9% of fourth graders (n=225) indicated performing service the same year.These results may point to a tendency for schools to incorporate service projects in the classroom at certain grade levels more than others.
Factor analysis enabled us to combine multiple questions in the survey into a few core measures of student attitudes and behavior.The measures we constructed from the factor analyses included: Engagement, Good Behavior, Being a Good Student, and Helping Others.These measures were constructed based on the survey questions as described in the Technical Appendix accompanying this report.Correlations are reported in the tables below: Engagement and Good Behavior were the strongest of these measures with alphas of 0.87 and 0.71, respectively.Being a Good Student (alpha=0.63)and Helping Others (alpha= 0.58) were slightly less powerful measures but still provided a high enough correlation of survey questions that we felt confident in including them in our analysis.
We tested each of these factored measures against service outside of school, school, gender, grade, and service last year.The relationships between these factors and our measure of student engagement, presented in Table 2 below, demonstrated a significant relationship with service outside of school (r = -0.10),gender (r = 0.25), and grade (r = -0.16).Our study found that students who have participated in service projects outside of school are more engaged with their schoolwork, as compared with those who have not.Similarly, females are significantly more engaged with their schoolwork than are their male counterparts.The negative value for the measure for grade indicates that students in fifth grade are significantly less engaged in their schoolwork than are younger students.There was no significant relationship between engagement and the school and service last year variables.The figures in Table 3 below examine the relationship between the same set of factors and the composite measure of good behavior.The figures in the table show several significant relationships, such as that with gender, indicating females report better behavior in school than their male classmates.In addition, service outside of school is significantly related to behavior in school, with those participating in service projects outside of school having better behavior than those who did not.Similarly, participating in a service project in school in the previous year is positively and significantly associated with good behavior.As with student engagement, student's grade in school is negatively associated with behavior, indicating older students have worse behavior than do younger students.School was not significantly correlated with engagement.Helping Others was significantly correlated with three of the five tested variables.Gender and Helping Others were positively related (r = 0.23), as well as service outside of school (r = 0.12) and service last year (r = 0.12) both demonstrated positive relationships with Helping Others.The outcome examined in Table 6 is an assessment of how strongly the student is oriented toward others and would prefer to do things with and for others, rather than by and for him/herself.All four of the predictors are significantly related to the measure of being oriented toward others.Both service outside of school and in the past year are positively related to an orientation toward others.Females are significantly more likely to have an orientation toward others, relative to their male peers.The negative coefficient for grade indicates that older students are less oriented toward others than are their younger counterparts in school.In summary, it is interesting to note that gender was the only variable which was both significantly and positively correlated with all of the five factored measures.In addition, the correlation coefficients were quite similar for gender and the factors: Engagement (r = 0.25); Good Behavior (r = 0.21); Being a Good Student (r = 0.27); and Helping Others (r = 0.23).
In order to further examine whether significant differences existed between schools in the sample, we tested the intra-class correlation (ICC) between the factored variables and the school variable.The ICC value indicates the percentage of the outcome's variance which is attributable to between-school differences.Engagement, Good Behavior, and Help Others all returned significant differences between the nine study schools, but Being a Good Student did not illustrate any significant difference.Help Others showed the largest percentage betweenschool difference at 2.1%, while Good Behavior showed a 2.0% difference.The smallest difference occurred in the Engagement variable at 1.4%.These low percentages indicate that the students' responses to the factored survey questions were quite similar overall.This result also highlights the fact that although students in the study schools seem to share similar levels of Engagement, Good Behavior, Being a Good Student, and Helping Others, there are other factors driving the differential service participation between the schools.School variation was small when comparing school with the factored measures of students' attitudes and behavior, but school was strongly correlated with students' participation in service outside of school and service last year.

Results -Spring 2006
In the late spring of 2006, we re-interviewed students in the same grades in the same schools using the same survey instrument.Results from the data collected through these interviews are presented in this section.As with the data from the fall interviews, we created combination measures of student responses using factor analysis, then looked at the relationship between each of these factored measures against service outside of school, school, gender, grade, and service last year.The relationships between these factors and our measure of student engagement, presented in Table 7 below, demonstrated a significant relationship with, gender (r = 0.25), grade (r = -0.16),and service last year (r = 0.24).That is, students who have participated in service projects outside of school are more engaged with their schoolwork, as compared with those who have not.The correlation between gender and engagement is much larger in the spring data than in the fall (r = 0.51 in the spring, 0.25 in the fall).Interestingly, the variable for service outside of school, which was significantly related to engagement in the fall interview, is not related to engagement in the spring.The figures in Table 8 below examine the relationship between the same set of factors and the composite measure of good behavior.The figures in the table show several significant relationships, such as that with gender, indicating females report better behavior in school than their male classmates.In addition, service outside of school is significantly related to behavior in school, with those participating in service projects outside of school having better behavior than those who did not.Similarly participating in a service project in school in the previous year is positively and significantly associated with good behavior.As with student engagement, student's grade in school is negatively associated with behavior, indicating older students have worse behavior than do younger students.As with the variables shown in Table 7, the correlations between good behavior and each of the measures is greater than was the case in the fall.The magnitude of change varies, though most correlations are at least twice their levels of those in the fall data.Also similar to the relationships observed in the fall data, the measure for helping others was significantly correlated with three of the four tested variables.Gender and Helping Others were positively related (r = 0.23), with females more likely to be oriented toward helping others.Both of the service variables are positively and significantly related to Helping Others.The Being a Good Student variable only showed a significant relationship with gender (r = 0.58), and this relationship was positive with a correlation approximately twice that of the relationship observed in the fall data.Service outside of school, grade, and service last year were not significantly correlated with Being a Good Student.11 is an assessment of how strongly the student is oriented toward others and would prefer to do things with and for others, rather than by and for him/herself.Three of the four predictors are significantly related to the measure of being oriented toward others, with grade in school the only non-significant.Both service outside of school and in the past year are positively related with an orientation toward others.Females are significantly more likely to have an orientation toward others, relative to their male peers.Overall, the levels of outcomes stayed constant or declined somewhat between fall and spring.Some measures, such as community orientation or orientation toward others, changed very slightly between the two interviews.Student engagement declined over the interval more substantially.
These overall differences mask some interesting differences within categories of students.One important dimension of difference is gender, as shown in Table 13 below.For each of the outcomes examined, females have a higher level at both points in time.Another important dimension of difference is the grade level of students.Overall, our measures of student outcomes reveal that fifth graders generally have lower outcomes than do students in other grades studied.They are less engaged in school, exhibit worse behavior, and are less oriented toward their communities and others.The figures presented in Table 14 below document these differences.On all measures, third graders had the most positive outcomes, while those of fifth graders are lowest.However, there are important differences between schools in the outcomes of fifth graders.The students of this grade in some schools have very different levels of each of the outcomes.In the table below, School A is a school where the Penny Harvest program seems to be working very well, School B one where it is not working as well, and School C is a wellfunctioning program with a student population with a high level of poverty.

Explication of Results
The analysis reveals that this sample of New York City public elementary school students is highly involved in service projects, both inside (over 80 percent) and outside (over 40 percent) the school system.Such participation in service activities has been linked to a myriad of positive effects spanning civic, social, and academic realms.As expected, involvement in service projects both inside and outside of school was correlated with good behavior, helping others, and being oriented towards others, in both the fall and the spring semesters.Correlations with engagement were also evident, albeit less consistent.Being a good student is the only variable that was not correlated with service involvement.In general, the spring 2006 results were much more robust than the fall 2005 results.
Despite the correlations reported above, levels of engagement, good behavior, helping others, being a good student, and having an other-orientation were very similar in both the fall and the spring.One explanation for this is that it may be difficult to attain changes over such a short period, an explanation that suggests the need for longer term studies of this nature.That is, the effects of participation in the Penny Harvest program may be evident over the longer term of children's development, a time frame that would make it difficult to detect effects through this study design.It could also be that the similarity from one time point to the next may also reflect a ceiling effect.This sample of students already exhibited high levels of all factored measures in the fall.
The similarity of levels of beneficial and positive beliefs at both points may also result from the research design itself -particularly from the strategies used to select schools.In selecting schools in which the program had been running successfully, the design may have inadvertently screened out schools where the program's benefits may be most profound.The most dramatic effects of the program are most likely to occur at the outset of program implementation.The present sample only studied schools in which the program was already well-established, and had been in operation for at least two previous years.Indeed, as the participation results illustrate, the large majority of these students were already highly involved in service activities.
The fact that outcomes remained high, however, speaks to the effectiveness of the Penny Harvest program, and the importance of schools' continued involvement with service learning.Past research has indicated that, over time, even strong effects of the best implemented programs disappear at follow-up (Melchoir, 1997).
Demographic Differences: Schools, Grades, Gender While different schools exhibit different rates of participation in service activities, even the school with the least amount of reported service, still attained participation rates of nearly 70%.Participation outside of school was substantially lower, but the range of differential rates between schools was similar.There was little between school variation with respect to engagement, good behavior, helping others, and being oriented towards others, and virtually no variation with respect to being a good student.This may indicate that despite variations in the program between schools, the core aspects of this service learning initiative, shared by all schools, bear the most impact on students.
Differences between grades were minimal in terms of the amount of service activity performed outside of school, with younger participants only slightly more active than older students.Within school, 5th graders were most actively involved, followed by 3rd graders and then 4th graders.Interestingly, younger students were more engaged in their schoolwork and exhibited better behavior.They also demonstrated more of an other-orientation in the fall.This demonstrates that perhaps the program has a stronger impact on younger individuals.Since most research has not explored this elementary-school age-group, it is especially intriguing that it is the youngest individuals in this sample that are most affected.This is consistent with extant literature that demonstrates that younger individuals learn and adapt more quickly than older individuals (Hamilton & Fenzel, 1988).
Females participated in more service-based projects, both in and out of school, than their male counterparts.Females also showed significantly higher outcomes on all five composite measures, at both time points.This gender difference has consistently been found in past research (Hamilton & Fenzel, 1988;Scales, Blyth, Berkas, & Kielsmeier, 2000), and lends further support to the relationship between participation in service learning and various positive outcomes.

Limitations and Directions for Future Research
The present study addresses some of the gaps in previous research.This study provides more rigorous quasi-experimental designs in the form of both between-subject and within-subject comparisons, a review of multiple sites using the same program, a large sample size comprising a broad range of students, and an investigation of elementary school children, a younger population than is typically studied.Just as the strengths of this study stemmed from the limitations of past research, so too can the drawbacks of the present work guide future development in this field.
However, this exploration was preliminary and demands more research among this age group that will approximate true experimental research and provide conclusive evidence of the value of this treatment among young students.
The lack of appropriate control schools without the Penny Harvest program, prevent clearer estimates of the program's true effects.Instead, correlational analyses were performed, relating varying participation within Penny Harvest schools with varying positive outcomes.Causational inferences cannot be extrapolated from these correlational results.While increased servicelearning may engender increases in engagement and other positive outcomes, it is also possible that students who are more engaged with their schoolwork are more likely to participate in service activities.Furthermore, a third factor, such as service involvement outside of school, could be the cause of both increased participation in the Penny Harvest program and increases in the outcome variables.These extraneous variables should be controlled in the pure evaluation of the program.All of these issues are addressed with the use of a pretest-posttest design.The temporal ordering of the tests controls for reverse relationships, since increases in engagement at a later date cannot yield participation in a program at an earlier date.Additionally, the within-subject design controls for extraneous variables that could differ between subjects.
No significant pretest-posttest changes, however, were observed.As previously discussed, this most likely indicates a ceiling effect.Since this sample of students was already highly involved in service learning, they already had high outcome ratings at pretest.It is also possible that the short interval between time points played some role.Future studies could implement a pretestposttest design in schools with recently implemented programs, taking pretest measures before implementation and posttest measures after one year or more.Such a study would also serve to further increase the generalizability of the effects of the program.
Lastly, while the questionnaires used are valid and reliable instruments, there are certain drawbacks of using self-report measures.One such weakness is the social desirability bias, whereby people are motivated to present themselves in a positive light.An additional issue of comprehension may arise when dealing with such a young population.Future studies could employ behavioral measures, such as students' grade point averages and teacher's reports of children's conduct.

Implications
In addition to providing directions for future research, this study illustrated the various positive effects of service learning on elementary school children, and elucidated differential effects with respect to gender and grade.Increased understanding of the structure and effects of the Penny Harvest program will serve to inform the improvement of current programs as well as the implementation of future programs in New York City and elsewhere.Such research ultimately aids students, schools, and communities alike.

Table 4
The Being a Good Student variable only showed a significant relationship with gender (r = 0.27), and this relationship was positive.Service outside of school, grade, school, and service last year were not significantly correlated with Being a Good Student.

Table 11
The first question addressed is whether and to what extent student measures changed between the fall and spring interviews.Measuring and assessing these changes is an essential first step in examining what kinds of effects the program might have over the course of the year.Table12below compares levels of student outcomes for each of the five outcomes examined.

Table 12
Comparing Levels of Student Outcomes in Fall and Spring

Table 13
Comparing Levels of Student Outcomes in Fall and Spring, by Gender

Table 14
Comparing Levels of Student Outcomes in Spring, by Grade

Table 15
Comparing Levels of 5th Grade Student Outcomes in Spring: Three Schools Though this examination of the Penny Harvest program was performed only in New York City, its design has been carefully created to allow comparison between it and other Penny Harvest programs in different cities.Additionally, the careful modeling of the survey on Berkeley's Civic and Social Responsibility Survey will allow further comparison between this evaluation and other evaluations using the Berkeley survey.Thus, we compared responses for students in New York City with those from urban areas in which the Berkeley study has been used.This comparison suggests that the Penny Harvest program outperforms other similar service learning initiatives.The questionnaire measures 'self interest versus community interest' as well as 'altruism.'The first component includes items such as "I would rather spend time on my own activities than help someone else learn something" (reverse scored) and "It's important for all students to help out their school or community."The aggregate measure of altruism includes questions such as "I share things with others" and "I cheer up people who are feeling sad."Both composite measures exhibit validity and reliability.T-tests revealed that our sample was significantly more interested in their community (t=32.69,p<0.001) and significantly more altruistic (t=12.12,p<0.001) at posttest than the Berkeley sample.Unfortunately there are few other studies to which we can compare results indicating a great need for further study.