Breaking it Down : Defining the Framework & Evaluating the Impact of a Peer Education Program

Evaluation in the field of youth development continues to evolve. Youth development programs vary significantly in their focus, setting and outcomes. Community-based programs seeking to create or strengthen their evaluation methods and tools may have difficulty identifying what to measure and how to capture anticipated outcomes. This article focuses on a youth development program combining service learning and peer education, serving urban adolescents ages 14 to 19. The purposes of this study are: 1) to illustrate a strategy used to clarify and align core activities, anticipated outcomes and evaluation tools, and 2) to provide an overview of the updated data collection instruments created by the program.


Introduction
Crozer-Chester Medical Center's Peer Leader Program operates in Chester, PA, a mid-sized city just outside of Philadelphia.The goal of the program is to combine service learning, peer education and the Social Development Model (Hawkins & Weis, 1985) as a means to increase protective factors and decrease risk taking behavior among 14-19 year olds.The Social Development Model asserts that the most important units of socialization (family, schools, peers, and community) influence behavior sequentially.Positive socialization is achieved when youth have the opportunity within each unit to be involved in socially acceptable activities, when they develop skills necessary to be successfully involved, and when those with whom they interact consistently reward desired behaviors.These conditions increase attachment to others, commitment to conforming behavior, and belief in the conventional order.These social bonds to conventional society inhibit association with delinquent peers and, in turn, prevent delinquent behavior.
Crozer's Peer Leader Program is based on research indicating the value of the Social Development Model (Hawkins & Catalano, 1994), civic engagement and service learning (Roehlkepartain, 2007) and peer education (Mason, 2003).Established in 1996, the Peer Leader Program operates within a youth development framework, emphasizing building actual and potential strengths (developmental assets) of youth as opposed to making prevention of risk behavior the primary focus (Roth & Brooks-Gunn, 2003).
An extensive body of research has shown that certain risk factors are associated with the likelihood that teens will engage in risky behaviors related to violence, sex, and alcohol/tobacco/other drugs (Catalano, 2007;Centers for Disease Control and Prevention, 2008;Hawkins & Weiss, 1985;Kirby, 2007;National Institute on Drug Abuse, 2008;RMC Research Corporation, 2007;Roehlkepartain, 2007;Substance Abuse and Mental Health Services Administration, 2009).These factors include growing up in a poor community, having little attachment to one's neighborhood, low commitment to education, and friends who engage in risky behaviors.
Young people in Chester have significant exposure to these risk factors.The median household income in 2000 ($25,703) was about half of the county's median income.There are no youth-oriented leisure facilities and few local businesses provide part time jobs for youth.In 2004/2005, 53.5% of Chester High School graduates reported that they were planning to enroll in a two or four-year degree granting college or university or specialized associate degree-granting institution, compared to 77% of students from Delaware County and 71.8% of statewide students.Although Chester has 7% of the county's population, it represented 22% of total county arrests for juvenile aggravated assaults and 33% of total county arrests for juvenile weapons offenses in 2005.In 2000, the birth rate in Chester (79.6 for 15-17 year olds and 105.1 for 15-19 year olds) was three to four times higher than statewide and county rates.
Peer education programs such as Crozer's help to build resilience to risk factors while honing the ability of participants to become agents of change, creating a sense of empowerment, ability and connectedness (Pearlman et al, 2002;Ward, 2007;Zacharatas, Barling & Kelloway, 2000).High school students must successfully complete 40 hours of training to become Crozer Peer Leaders.Following the initial training, participants attend monthly enrichment and/or supplemental training activities.Peer Leaders work in teams of three to four under the supervision of adult staff, presenting lessons on life skills and risk reduction to middle school students.Lessons are presented in the public school classrooms during the school day or in community-based after-school programs.The average student takes part in over 70 hours of programming per year and most remain actively enrolled for several years.
Crozer's Peer Leader Program has grown organically over time.While the basic structure of the program has been consistent, the program has been modified and refined over the years in response to emerging research, internal program evaluations, input from participants and funding.Staff observations and participant self-report indicated that the program was positively impacting youth, but with limited funding for evaluation we struggled for years find valid measures that were scientifically sound and would adequately capture our targeted outcomes.This struggle is shared throughout the field; other youth development practitioners have noted little consensus around what constitutes the core elements of youth development programs and what outcomes to measure (Ferrari, Arnett & Cochran, 2007;Sabaratnam & Klein, 2006;Catalano et al, 2004).Through participation in the federal Substance Abuse & Mental Health Services Administration's "Service to Science" initiative, we were able to contract for technical assistance from a team of seasoned evaluators to guide us through an extensive process of defining categories of program activities, anticipated dosage of each activity category, anticipated short-, intermediate-and long-term outcomes, assessment tools, and our annual evaluation timeline.

Development of the Evaluation
In order to develop a comprehensive logic model and evaluation plan for Crozer's Peer Leader Program, our first step was to convene focus groups with current and past Peer Leaders and program staff to identify commonalities in perceived areas of program impact; we wanted to ensure we didn't overlook any areas of impact that were not part of the program's original conceptualization.(For example, until youth reported gains in workforce skills such as interviewing and punctuality, staff had not considered formally measuring this aspect of the program.)Simultaneously we reviewed the literature on risk and protective factors linked to substance abuse, violence, drop-out and teen pregnancy to define our primary, vs. ancillary, foci.Although the program addresses many risk and protective factors to some degree, only those being addressed with ongoing, full-scale activities were included in the logic model and evaluation: 1) Antisocial beliefs and attitudes toward target behavior; 2) Level of community participation/engagement; 3) Academic performance, attendance & educational expectations; and 4) Connection to pro-social peers and adults.
Next we considered activities used to address targeted risk and protective factors.We reviewed all program activities from the past year and agreed upon six categories to capture them: 1) Core Peer Leader Training-essential knowledge and skills required in order to perform Peer Leader role; 2) Supplemental workshops-topics not required in order to perform Peer Leader role, but beneficial to Peer Leaders themselves (such as financial management or career workshops); 3) Service Learning-presentation of life skills & risk reduction workshops to younger adolescents; 4) Team building activities to promote bonding with peers and adult staff; 5) Recognition events; and 6) Enforcement of program requirements and standards (i.e.being punctual and prepared, maintaining a grade point average 2.5 or greater).
We then considered short-, intermediate-and long-term outcomes and data collection points for each activity area, being mindful not to overburden staff or participants with data collection activities and thus jeopardize accuracy of the data.We settled on 8 outcomes and the time frames for assessing each.(See Appendix A-logic model.Also see Table A, "Related Outcomes" column.) For some outcomes there were well-established measures.In most cases these assessments were too long to be tolerated by participants if used in their entirety.Thus, we extracted the most relevant sub-scales.For other outcomes we identified instruments that required minor modification.And still for others we were unable to identify any measures that adequately captured the outcome and had to create our own assessments.Service learning and leadership aspects of the program were by far the most difficult to capture, as the instruments we reviewed for possible use didn't capture the aspects of service and leadership stressed within the program.
Throughout the process of developing instruments we were mindful of the need to customize for our population, students within an urban, primarily African-American school district with average reading levels well below grade level.This had implications for length, phrasing, tone, layout and protocol for survey administration.As we developed the instruments to be used we sought feedback from staff and contacted past participants, which helped us to fine-tune the instruments and process.The feedback made it clear that we would need to make items on risk taking behavior anonymous, not merely confidential, in order to get truthful answers.
Nearly everyone felt that social desirability would be an issue-that participants would "tell you what you want to hear."Ultimately we devised an evaluation plan that included 8 evaluation tools.Please see Table A for tools, related outcomes for each tool, and source of tools.

Discussion
It would seem that, like ours, most youth development programs are modified over time based on funding, research and staff/participant suggestions.Distinguishing between core and supplemental training and the anticipated outcomes of each allowed us to determine which aspects of the program needed to remain constant and which could be modified.For example, core training for our peer educators needs to remain consistent from year to year so that they are prepared to adequately facilitate workshops.However, supplemental training topics can shift year to year yet still lead to improved knowledge and attitudes related to life skills and adolescent risk taking behavior.Our data collection instruments were designed to work yearto-year regardless of modifications.For example, the questions used to assess connection to pro-social peers and adults will still be appropriate if we change team building activities.
Similarly, the pre/post workshop assessment forms used with each supplemental workshop have a standard format wherein participants rate their knowledge or attitudes on workshop content before and after the workshop.This format provides us with a standard instrument to measure pre/post change, yet with the flexibility to select topics to meet changes in needs and interests of our participants.We believe modifications are important for continuous quality improvement but acknowledge that they would be a potential limitation on a multi-year study, given that content and dosage would not be constant.
Qualitative data provides evidence that participation in this program changes how adolescents define themselves (leaders, role models), how they feel the community views them, and how they feel about service.They report being stopped on the streets, in the mall and other public places by middle school students they have taught via peer education sessions and they report a positive pressure to "walk the walk" not just "talk the talk".In addition, they report not wanting to engage in behavior that will hurt the reputation of the group and tarnish the reputation of Peer Leaders as a whole.We hypothesize that these items are among the most critical in leading to behavior change.However we found it difficult to identify existing instruments to address these items; most tools we found related to civics focused on items such as voting or knowledge of political process.This may be because service learning and peer education programs are only a fraction of the greater youth development / prevention field and that the body of literature is smaller and still emerging.A thorough literature review and investigation would be of benefit.
Practitioners are faced with a paradox, trying to balance the need to use established data collection instruments to make claims about program success and the need to customize instruments for the program and population.The danger in relying only on established data collection instruments is that the program risks measuring the wrong concepts and not actually capturing the benefits being provided by the program.Such mis-measurement can create an inaccurate picture that the program is not performing well.We attempted to find balance in this paradox by drawing from established instruments when feasible, but recognized that some measures would have to be developed internally and tested for reliability and validity moving forward.
We also made decisions about anonymity vs. confidentiality based on participant feedback, opting to separate the participant survey into an anonymous portion for risk taking behavior and a confidential portion for all other items.While we will lose the ability to link individual responses over time on risk behavior items, we will have greater confidence in the validity of the data.

Implications
There is growing consensus in the field of youth development about the objectives common among positive youth development programs.Some of these objectives include promoting bonding, fostering resilience, promoting cognitive competence and fostering positive identity (Catalano et al, 2004).Despite consensus on objectives it remains difficult for practitioners to identify data collection instruments that are appropriate for their population and that truly capture the spirit of the activities occurring in the program.
Youth development practitioners and evaluators are encouraged to consider how to move the field forward with respect to: 1) The need for balance between tested vs. tailored data collection instruments; While instruments tested for reliability and validity and used effectively with similar populations are ideal, they may not assess a program's unique characteristics.In addition there is the risk that practitioners will modify their programs to focus on activities and outcomes that are "measurable" with tested instruments as opposed to what the youth truly need; 2) Technology-informed methodology; There is a need to incorporate data collection methods that are youth-friendly (i.e.web-based surveys, hand-held participant response systems, use of social networking platforms to poll youth, etc) to reduce youth from feeling data collection fatigue and; and 3) Modular approaches to data collection; Youth development practitioners would benefit from a compendium of short, tested evaluation tools that can be used in an a la cart fashion to assess objectives common among positive youth development programs.Having the ability to pick and choose from a variety of short, tested instruments would allow practitioners to create evaluation plans tailored to their program's primary objectives yet not cumbersome in length and still have scientific rigor.In addition, modular data collection would allow programs to add or remove sections of the evaluation in the event of increased or reduced funding, while leaving remaining sections in tact.
These suggestions would enhance the ability of program practitioners to measure anticipated outcomes of youth development interventions in a way that is easily incorporated into the multifaceted environment of community based programs with limited resources and changing populations.

Table A
Data Collection Tools, Related Outcomes, & Sources of Tools for Crozer-Chester Medical Center's Peer Leader Program Logic Model: Crozer-Chester Medical Center's Peer Leader Program Problem/Need Statement: Chester youth experience many of the community, family, school, and individual risk factors associated with substance abuse, teen pregnancy, interpersonal violence, and drop-out.There is a need to engage youth in programs that will build protective factors, which will help to insulate them from the potential negative impact of risk factors Goal: To implement a youth development program combining peer education, service learning and the social development model as a means to increase protective factors and decrease risk taking behavior among 14-19 year old participants of Crozer-Chester Medical Center's Peer Leader Program, in the urban community of Chester, PA.