What Counts in After School ? Findings from the Massachusetts Afterschool Research Study ( MARS )

This article discusses the Massachusetts Afterschool Research Study (MARS). Conducted during 2003-2005, MARS took an in-depth look at program structure and quality in 78 varied programs across Massachusetts, using data sources that included interviews with program directors, afterschool program site observations, school district student data, attendance data, and surveys with afterschool program staff, day school teachers, and afterschool program youth. The MARS study offers many useful insights into what afterschool programs look like, approaches to providing high quality experiences for youth, and the connections between high quality and improved outcomes for the young people attending these programs. The results may be useful to programs, policy makers, and others in the field by deepening our understanding of how youth participation leads to a variety of youth outcomes.


Introduction
We know that simply having an afterschool program in place is not enough to result in positive outcomes for youth, yet there are very few studies to guide the creation of programs that will have their desired results.This paper discusses the Massachusetts Afterschool Research Study (MARS), one of the first studies to open the "black box" of afterschool program practices.By looking in depth at program structure and quality in 78 varied programs across the commonwealth of Massachusetts, MARS helps us to understand the links between program participation and youth outcomes.
The Massachusetts Afterschool Research Study had two major goals: (1) to identify those program characteristics that are most closely related to high quality implementation, and (2) to explore the links between program quality and youth outcomes.Several previous studies guided the MARS research design.
In two studies of elementary school children, Vandell and her colleagues (Pierce, Hamm, & Vandell,1999;Rosenthal & Vandell, 1996) found that positive interactions between staff and children, as well as between children and peers, were related to successful functioning.A team of researchers from the RAND corporation (Beckett, Hawken et al., 2001) reviewed all existing studies of afterschool program quality in 2001.While concluding that there were few studies emulating high scientific standards, the RAND researchers found a number of program practices that the data supported as good indicators of program quality, including a high level staff training, education, and compensation, low child-to-staff ratio, age-appropriate activities, positive emotional climate, communication with school and families, and community partnerships (Pierce, Hamm, & Vandell, 1999).
Common sense suggests that different programs may lead to different outcomes.For example, one program might be particularly effective in promoting math skills, while another builds children's motivation to learn, and a third increases their ability to get along with peers.By examining a range of academic and non-academic outcomes, and linking these to program practices, the MARS study was designed to build an understanding of the complex relationships between program goals, program practices, and outcomes for youth.

Conceptual Framework for the Study
The MARS study was conducted in a total of 78 afterschool program sites in ten geographically and economically diverse school districts.Building on previous studies (Beckett, Hawken, & Jacknowitz, 2001;Eccles & Gootman, 2002;Reisner, White, Russell, & Birmingham, 2004), the study researchers developed a conceptual framework that posits two types of program features: (1) program characteristics, which vary between programs, are structural in character, and may or may not be related to program quality; and (2) program quality, which is process-oriented, and captures the actual program as it is implemented.
Program characteristics affect program quality, as well as leading to outcomes for youth (see Figure 1).Other factors also affect youth outcomes.Child and family characteristics such as income, gender, race, or special needs status might affect the extent or ways in which program participation results in youth outcomes.In addition, simply enrolling in a program is not likely to lead to outcomes unless a child spends a significant amount of time there (intensity) over a number of months (duration).The effects measured by the MARS study include both academic and non-academic outcomes.

Methods
In order to answer our research questions, we collected information on program characteristics as well as the individual children and youth who attend the programs.For example: Furthermore, we collected data on school attendance from school districts, program attendance from programs themselves, and child and family characteristics from both programs and schools.

Program Quality
The quality of the MARS programs was primarily collected using the Assessment of Afterschool Program Practice Tool-Research Version (APT-R).The APT-R has three sections: Overall Program Observation; Activity Observation; and Homework, each with a number of items on a four-point scale ranging from "not true" to "very true."We summarized the results of the APT-R observation items into five key Quality Indicators.Each of these quality indicators summarizes a number of items from the APT.These items were chosen as the strongest indicators of quality after extensive testing through both factor analyses and scale analysis (i.e., Cronbach's alpha).The Quality Indicators are listed with examples of the key items that were combined through their factor loadings for the analysis.

Youth Outcomes
To capture youth outcomes, we utilized the Survey of Afterschool Youth Outcomes (SAYO).
The SAYO is a brief survey of questions about an individual student, conducted in the fall and again in the spring.It includes a version completed for each youth by an afterschool staff person (SAYO-S) as well as a teacher (SAYO-T) and collects information on a range of both academic and non-academic outcomes.Complete information on this instrument is reported elsewhere (Miller & Surr, 2002).

Sample
The diverse statewide sample of 78 programs in the MARS study included 58 programs located in urban areas, 14 programs located in suburban areas, and 6 programs located in rural areas or small towns.The following 10 Massachusetts public school districts collaborated with the MARS data collection: Boston, Fall River, Framingham, Greenfield, Holyoke, Lowell, Northampton, South Hadley, Turners Falls, and Worcester.
The study included a wide variety of programs.Programs received funding from their local United Way, the Office of Child Care Services (OCCS), and the Department of Education's (DOE) 21 st CCLC grants, with some programs receiving funding from more than one of these sources (see Table 1).Sixty of the 78 programs in the study exclusively served elementary school youth, 15 served only middle school youth, and three had youth ranging from grades K-8.A small percentage of programs served youth in grades 9-11.The study included programs operated by schools, YMCA's, Boys & Girls Clubs, Citizen Schools, faith-based organizations, and many other organizations.While the sample of programs selected to participate in MARS is very diverse, it does not include programs with a single-area focus, such as sports, academic remediation, or the arts, and does not include drop-in or high school programs.The 78 programs selected for MARS serve a diverse population of youth that closely mirrors the diversity of the school population in the cities and towns where these programs operate.Overall the afterschool programs sampled in MARS served a youth population that was 46% White, 26% Hispanic, 21% Black, 6% Asian, and 1% other ethnicity.The afterschool programs in the MARS study utilized a range of paid and unpaid staff, including community volunteers, high school student tutors, regular school day teachers, and various specialists.The overall profile for staff in the MARS study was: 66% White, 16% Hispanic, 15% Black, and 3% Asian.The number of paid staff in programs ranged from 2 to 35 staff.Across the whole sample of programs in MARS there was an average of 8.6 paid staff per program.The data suggest that having smaller group size for activities was related to higher staff engagement-probably because staff have fewer children with whom they need to communicate and connect-as well as with activities that are more challenging and engaging.Also, programs with engaging, challenging activities had: smaller group size, a higher percentage of time in structured activities, fewer days per week in homework time, and stronger connections with the school.
A few program characteristics were associated with higher quality on some indicators and lower quality on others.Programs that had more children enrolled, as well as those that spent more time in structured activities during a typical afternoon, were more likely to have engaging, challenging activities, but demonstrated less communication with parents at pick-up time.
The Staff Engagement Quality Indicator was linked to a number of program characteristics.We observed more engaged staff in programs with smaller groupings of youth for activities, stronger connections with their feeder or host schools, a good pace to their well-organized schedule, and those serving middle school youth.Youth Engagement was also linked to some program characteristics.Youth were more engaged in programs with a well-paced schedule, where they were not being rushed from one activity to another.In addition, well-organized programs with clear routines had higher youth engagement.
The MARS study also collected information on the interactions between programs and families.
Programs that had better communication between staff and parents at pick-up time ("Family Relations") also tended to report stronger connections with parents and the community in the director interviews.Programs that exhibited connections at pick-up time were significantly more likely to serve elementary school youth, have a smaller overall enrollment, be less structured, and have a schedule that allowed children to move at their own pace.

Staff Characteristics and Quality
Previous research in afterschool and education has pointed to the importance of program staff in providing a high quality experience for youth (for example, see Commission on Children at Risk, 2003;Pianta, 1999;Rosenthal & Vandell, 1996;Thompson & Kelly-Vance, 2001).Therefore, it is not surprising that a number of staff characteristics in the MARS study had significant relationships with program quality (see Table 3).Programs with more highly educated staff, both at the program director and direct service levels, were rated significantly higher on program quality, including staff engagement, youth engagement, activities, and homework time.In addition, programs that utilized certified teachers and other school staff tended to rate higher on these Quality Indicators.
In addition to background characteristics, the working conditions of the staff were associated with higher or lower quality in a variety of areas.Higher wages are linked with higher quality in all areas except communication with families and more training is related to higher quality staff engagement.Higher staff turnover (more staff leaving during the school year) is associated with lower quality ratings in both youth engagement and homework time.Programs in the MARS study generally had low staff: youth ratios (i.e., 1 staff person per 8.4 youth, on average).The results in Table 3 indicate that the programs with lower ratios are also implementing higher quality programs, as indicated by staff engagement, youth engagement, activities, and homework indicators.These results are congruent with those in previous literature on afterschool programs as well as extensive research in the early childhood field (for example, Beckett, Hawken et al., 2001;Clements, Reynolds et al., 2004;Helburn, 1995).A low staff: youth ratio enables staff to be able to build relationships with young people, individualize their facilitation of activities, and work individually or in small groups as needed.

Program Participation and Youth Outcomes
In the previous section, we discussed the relationship between program characteristics and program quality.The MARS study also asks: What difference does quality make for youth?To answer this question, we used the Survey of Afterschool Youth Outcomes (SAYO).This measure collects pre and post data on a variety of academic and youth development outcomes of afterschool program participation that have been documented in previous studies (see Beckett, Hawken & Jacknowitz, 2001;Bodilly & Beckett, 2005;2001;Grossman, Price et al., 2002;Harvard Family Research Project, 2004;Huang, Gribbons et al., 2000;Kane, 2004;Mahoney & Zigler, 2003;Miller, 2003;Policy Studies Associates, 2001;Scott-Little, Hamann et al., 2002;Walker & Arbreton 2004;Warren, Feist et al., 2002;Witt, King et al., 2003).See Table 4 for a list of SAYO Outcomes.Over the course of the 2003-2004 school year, the average scores increased on all outcomes (individual children went down in their scores, but aggregated across programs, the results were positive).The greatest positive change was in teacher ratings of problem-solving and analysis skills, followed by teacher ratings of homework completion and quality, as well as staff ratings of initiative and relations with adults.
We used a fairly complex statistical approach, called multilevel modeling, to control for children's pre-test scores as well as a set of demographic characteristics.The goal of this approach was to separate out the influence of program quality/program characteristics from other possible causes of the change in SAYO outcomes.Our analysis indicated that, while change in SAYO-T scores (provided by teachers) varied by child background characteristics, they did not vary by program characteristics or program quality.In other words, while children's scores on the SAYO-T increased over the course of the year, we could not establish links between the changes in outcomes and the quality of the afterschool programs the children attended.
However, we did find connections between program quality and changes on the SAYO-S surveys completed by program staff.This difference between the teacher and staff ratings may be due to the closer relationship between outcomes observed by program staff and children's experiences in the program.For example, the quality of a program and children's behavior in the program are conceptually closely linked.Therefore, the models assessing the effects of program quality were tested for only those child outcomes rated by after-school staff on the SAYO.

Program Characteristics and Youth Outcomes
The past section looked at the connections between program characteristics and program quality, and found a number of significant relationships.Now we move on to see whether program characteristics had any direct relationship with youth outcomes.This is a more complex and distant relationship, since youth outcomes are likely to be affected by a host of other factors, including experiences in school, home life, among others.Table 5 depicts the relationship between selected program characteristics and youth outcomes measured by the SAYO-S.
In Table 5, the SAYO-S outcomes are listed in the columns across the top and the program characteristics are in the rows at the left.Up arrows indicate that more of the program characteristic is associated with a larger positive change in the youth outcome, that is, a positive relationship at a statistically significant level (p < .05).The results in Table 5 indicate that strong relationships with teachers and principals of a school, probably facilitated by location at the school, may be helpful in promoting positive outcomes for youth.Interestingly, some of our other measures of relations with schools, including the extent of communication with the school, having academic achievement, or homework improvement as goals, and having more methods for communication with schools, were not related to changes in youth outcomes.
Program location in a school also had a significant link to two youth outcomes-Relations with Peers and Initiative.While it is not shown in the table, program location was also related to Relations with Adults and Behavior in the program (at a level just under statistical significance, or p <.10).It is important to note that location in a school does not mean that a program is operated by the school.In the MARS study, most school-based programs were operated by other organizations, often community-based or national youth-serving agencies.Some program characteristics that had significant relationships to program quality were not linked statistically to youth outcomes.For example, the size of the program, staff-to-child ratio, and ages served (elementary or middle school) had no relationship to the level of change in youth outcomes on the SAYO-S.

Staff Characteristics and Youth Outcomes
We found that certain staff characteristics were related to higher observed quality in afterschool programs.In Table 6, up arrows signify a positive relationship between staff characteristics and youth outcomes; while down arrows indicate a negative relationship.Increases in children's homework persistence and completion were related to programs with more highly educated staff and directors, as well as those with lower staff turnover.Programs with lower staff turnover, a higher percentage of certified teachers, more staff with college degrees, and directors with higher levels of education had significantly higher quality in at least one of the SAYO-S outcomes.However, none of these factors seemed to play a major role in directly affecting youth outcomes, as we can see from the fact that they each affected only one outcome, and the relationship was at a fairly low level of statistical significance.Interestingly, improvements in homework seemed to be most affected by staff characteristics, as indicated in Table 6.

Program Quality and Youth Outcomes
We expected that high quality programs would be more likely to reach their desired outcomes for youth, and our results bore this out.Program quality explained between one-quarter and one-half (R 2 =0.27 to 0.47) of the variability across programs in youth outcomes, a relatively high level of explanatory power.Inter-factor correlation analyses of the relationships between the Quality Indicators showed strong relationships between youth engagement, challenging activities, high quality homework time, and staff engagement (see Figure 2).Relations with families and appropriate space were not so inter-connected with the other indicators.

Figure 2
Path A path model for each of the five SAYO-S outcomes indicated that appropriate space led to positive staff engagement, which in turn led to positive youth engagement, challenging activities, and high quality homework time.In every case, youth engagement was strongly linked to the SAYO outcome.
Except for youth engagement, the line from the quality indicators to youth outcomes was more indirect; challenging activities and quality homework were not directly linked to positive changes in youth outcomes in the MARS data set.This result is likely due to the lack of variation in our data in these areas.We saw only a few examples of challenging, engaging activities during our observations.Homework time was similarly lacking in variation-only a few programs scored either very high or very low in this area, resulting in a lack of statistical power to show relationships to the youth outcomes.Further research in this area may be helpful in identifying the links between quality activities or homework time and positive results for youth.
Family relations had a positive but not significant effect on four of the SAYO-S outcomes, and a positive, statistically significant effect on changes in Relations with Adults.This finding is notable in light of the importance of positive relations with adults in supporting youth success in education and development in general.We know from decades of research on resiliency that strong relations with caring adults are key to healthy development, including educational success (Blum, Beuhring et al., 2000;Commission on Children at Risk, 2003;Rutter, 1987;Werner, 1993).
The MARS findings suggest that youth engagement is key to positive outcomes for youth in afterschool programs.While other aspects of quality may promote youth engagement in the program, or in some cases, be directly linked to youth outcomes, we clearly need to pay particular attention to this Quality Indicator.

Conclusion
It is stating the obvious to say that no single study, taking place over the course of a single academic year, can provide definitive answers to the many questions we have about afterschool programs.On the other hand, at a time of increasing interest and investment, the results of MARS add to our understanding and form a foundation for future work in this area.We know that, in general, MARS supports the limited prior research in the field.Based on the results of the MARS study the following emerge as key issues for consideration: Workforce issues are important to quality.
• Staff make a difference in program quality, and staff who have a strong educational background and appropriate training are key to program quality.At the same time, given current wages and working conditions, many programs can not attract such staff, or even if recruited, lose a significant portion of staff each year (Dennehy & Noam, 2005).• A highly qualified coordinator or program director is key to setting the tone for a program that promotes youth engagement, staff engagement, and the quality of activities and homework.• Most programs in the MARS sample had very low staff-to-child ratios, typically between 1:7 and 1:9.We found clear links between low ratios and high quality, as has previous research in the field (Bodilly & Beckett, 2005).At the same time, where ratios are very low, programs may want to consider the trade-off between paying higher salaries and having additional staff.
Communication with families is connected to youth outcomes.
• One of the key youth outcomes-relations with adults-was positively associated with the quality of family relations we observed at pick-up time.Programs rated high on this component of the APT-R also tended to cite parent, community, and volunteer support as strengths of their programs.This finding suggests that all afterschool providers could benefit from paying attention to their relationships with the families of the youth in their programs.
Many programs could improve the quality of their activities.
• Most of the activities we observed tended to be short-term in nature and seemed only minimally related to program goals.Most did not have a clear connection to larger themes, curricula, or projects, and did not require higher level or critical thinking on the part of children and youth.While staff were usually available, they did not maximize their impact by asking questions, posing challenges, or extending the ideas of the children.We saw a range of programs, including some that epitomized many of these staff practices, but they were the exception rather than the rule.In addition, we did not see much evidence, either in materials or the activities, of an effort to integrate children's cultural and ethnic backgrounds into activities.Unless programs are intentional about the goals they are trying to achieve through their activities, they are unlikely to impart skills or develop new interests.Long-term projects allow children to achieve mastery, practice skills learned in school, make connections between different subject areas, and become engaged in their own learning process.Afterschool programs can play a powerful role as "border zones," creating meaningful connections between participants' cultural backgrounds and the mainstream skills and opportunities represented by schools (Heath, 1994).• We saw little evidence of practices that develop youth leadership skills, promote autonomy in youth, or support peer-to-peer learning strategies.This finding is supported by the perceptions of middle school youth, who gave programs their lowest ratings in the areas of providing opportunities to plan or lead activities.Programs should consider involving youth, including those in elementary school, in planning, choosing, and leading activities.Activities such as "book buddies" and the Design-It!engineering curriculum promote cooperative learning, thereby building both social and cognitive skills.
Relationships with schools are related to quality.
• Programs that reported strong relationships with teachers and principals were delivering higher quality activities, had high staff engagement, and provided higher quality homework assistance.For programs that want to increase the outcomes measured by the SAYO instrument, developing positive relationships with school personnel may play a role in successfully reaching their goals for youth.
Positive outcomes for youth are the result of a pathway leading back to youth engagement in the program, which in turn is a result of staff engagement with youth.
• Programs that want to support positive outcomes for participants need to have staff who can connect well with youth, interacting with them in a positive and respectful manner.
• Staff who engage with youth in positive ways will be supporting the engagement of participating youth in the program and with each other.Youth engagement is the strongest predictor of positive outcomes.
The MARS study offers many useful insights into what afterschool programs look like, approaches to providing high quality experiences for youth, and the connections between high quality and improved outcomes for the young people attending these programs.MARS is one of the only studies that examines the relationships between program characteristics and program quality, as well as identifying the key indicators that help produce positive outcomes for youth.Unlike previous studies that compare program participation and non-participation, MARS looks at the types of experiences and relationships that children have during their participation in afterschool programs, and how the quality of these experiences and relationships relate to youth outcomes.MARS also contributes to the literature by deepening our understanding of how youth participation leads to a variety of youth outcomes.
Conceptual Framework for MARS Showing Elements of the Study including Program Characteristics, Program Quality, Child and Family characteristics, Dosage, and Youth Outcomes Overall, the MARS study intended to answer the following research questions: 1.What kinds of afterschool programs are children and youth in the study attending?What variation do we find in program characteristics and features, including activities offered, ages served, goals, type of staff utilized, and level of participation by youth? 2. In what ways are program characteristics and features related to program quality? 3. What outcomes for youth are associated with attending afterschool programs?How do these outcomes vary depending on program practices and features?Which program characteristics and aspects of quality are associated with youth development and student learning outcomes?
We visited each program in the fall of 2003 and again in the spring of 2004, conducting interviews with program directors in the fall and interviews as well as observations in the spring.Information gathered through the Director interviews included information about:

Table 1
The average daily attendance for all 78 programs on the day of the May 2004 visit was 74 percent of total enrollment.Almost exactly half of the MARS program participants were boys and half were girls.The amount of time children participated at the afterschool program had a range of less than 50 to 850 hours per year.The average amount of time participating across all programs was 191 hours per year, or approximately 5.6 hours per week.Participation was higher for elementary school-age children than middle school-age youth.Programs that serve elementary school-age children are open longer -an average of 16.8 hours per week (3.4 hours per day), as compared with programs for middle school-age youth, which operate an average of 12.8 hours per week (2.6 hours per day).Programs that serve children in grades K-5 also tend to stay open longer -an average of 35 weeks/year, whereas programs that serve middle-school-age youth operate an average of 29 weeks/year.
Program Characteristics and QualityOne of the most important contributions of the MARS study is in helping to understand how various program characteristics-such as program size or connections with schools-are related to the quality of the program.In Table2, key program characteristics are listed in the rows, while the five Quality Indicators are arranged in columns.Up arrows indicate that more of the program characteristic is associated with a higher quality score on the indicator; while down arrows indicate lower quality.All arrows signify correlations that are statistically significant at the p < .05 or greater level.

Table 2
Selected Program Characteristics Associated with Program Quality Indicators* * All arrows signify correlations that are statistically significant at the p<.05 or greater level.

Table 3
Selected Staff Characteristics Associated with Program Quality Indicators* * All arrows signify correlations that are statistically significant at the p<.05 or greater level.

Table 4
Outcomes Measured by the Survey of Afterschool Youth Outcomes (SAYO)

Table 5
Program Characteristics and SAYO-S Youth Outcomes* * All arrows signify correlations that are statistically significant at the p<.05 or greater level.

Table 6
Staff Characteristics and SAYO-S Youth Outcomes* Arrows indicate statistically significant relationships at p<.05 or less. * model of the relationship between Quality Indicators and Youth Outcomes