Implementing and Evaluating the Communities That Care Prevention System in Rural Pennsylvania

Communities That Care (CTC) is a prevention-planning system that helps community members plan, implement, and evaluate evidence-based prevention strategies that target common behavioral problems among youth. Recent evaluations of the system have indicated promising results. To date, most of these studies have examined broad effects across multiple implementation sites. The purpose of the current study was to determine if such promising results could be replicated in a single rural county that has been implementing the CTC system for more than a decade. The results of this utilizationfocused evaluation indicate a variety of positive effects on outcomes among adolescents.


Introduction
Over the past three decades, the field of prevention science has grown dramatically, and has attracted the attention of numerous applied youth development professionals, including Cooperative Extension specialists and educators, human service agency representatives, government officials, university faculty, and many others.These professionals have become involved in the development, implementation, and evaluation of a wide variety of strategies aimed at preventing common adolescent behavioral problems such as delinquency, alcohol use, and school dropout (Author citation, 2010;Bogenschneider, 2006;Greenberg, 2004;Jenson & Fraser, 2006;Mincemoyer et al., 2008;Perkins, Mincemoyer, & Lillehoj, 2006;Small, 2005;Spoth, Kavanagh, & Dishion, 2002).
Since the early 1980s, prevention strategies have not only become more common, but they have evolved to become more sophisticated and more effective than their earlier counterparts (Catalano, Hawkins, Berglund, Pollard, & Arthur, 2002;Hawkins, Catalano, & Arthur, 2002;Olson, 2010).As a result, many local communities have chosen to invest resources into developing and/or purchasing existing prevention strategies, including curricula and various types of school, community, and family-based programs.
The Communities That Care Prevention System CTC is based on the public health risk and protective factor approach to prevention in which community members seek to reduce risks while enhancing protective factors.While this approach can be used to address a variety of problematic outcomes, most communities have focused on addressing delinquency, substance use, teen pregnancy, and school dropout (Hawkins et al., 2008;Hawkins & Catalano, 1992;Hawkins, Catalano, & Arthur, 2002).The planning system helps guide communities through the entire planning process, from start to finish.Specific resources include materials that help communities form collaborative coalitions, conduct needs assessments, prioritize risk and/or protective factors, select evidence-based prevention strategies, and implement and evaluate their effectiveness (Hawkins et al., 2002;2008;Hawkins & Catalano, 1992).

Effectiveness of CTC
Preliminary evaluations of CTC in Pennsylvania and other states indicate a variety of positive effects associated with program implementation.For example, CTC initiatives in Pennsylvania have been extensively evaluated since the commonwealth invested heavily in the system during the late 1990s and early 2000s.During that time, more than 100 Pennsylvania communities implemented at least part of the CTC system.A combination of process and outcome evaluations have indicated most of these communities were successful in implementing each of the components of the system and successful implementation was associated with positive outcomes (Feinberg, Bontempo, & Greenberg, 2008;Feinberg, Greenberg, Olson, & Osgood, 2005;Gomez, Greenberg, & Feinberg, 2005).Indeed, a large outcome evaluation revealed that adolescents living in communities that had implemented CTC experienced fewer risk factors and had fewer negative outcomes as compared to those living in non-CTC communities (Feinberg et al., 2005).

Limitations of Past Evaluations
A significant limitation of the above-mentioned outcome evaluation was that the effect sizes were extremely small.However, the consistency of the findings across multiple risk factors was impressive, and the evaluation provides at least preliminary data that the system may result in positive effects.More recent evaluations in other states have demonstrated similar positive outcomes (Hawkins et al., 2008).
One potential reason for small effect sizes in the Pennsylvania evaluation is the sheer size of the sample (79 school districts in 2001 and 147 districts in 2003) made it difficult for the evaluators to adequately determine the level of implementation within each of the CTC communities.As such, the definition of "CTC community" was broadly defined and included communities that varied widely in the level of implementation.Some had implemented as many as nine or ten prevention programs, while others had one or two.Some had targeted a wide variety of risk and protective factors, where others targeted a small number (Feinberg et al., 2005).The researchers attempted to control for such variation by conducting sub-group analyses in which only communities with "evidence-based" programs were included in the experimental group (about two thirds of the CTC communities).However, these analyses resulted in fewer rather than more statistically significant effects.In light of the relative imprecision due to variation across communities, the small effect sizes found in this study should not be a surprise, and it is rather impressive that any consistent effects were found.

Goals of the Current Study
In light of the wide variation in program implementation across communities in the Pennsylvania sample, the purpose of the study described in this paper was to examine one CTC community in Pennsylvania that has implemented a wide variety of evidence-based strategies.Specifically, we examined both implementation and outcome data from one rural county in central Pennsylvania that has had the CTC system in place since 1999.By examining data from a single county, we can complement the statewide evaluation and determine the degree to which a fully implemented system has had an impact on risk factors and outcomes among adolescents between 2001 and 2007.
The primary purpose of this study was to provide data that could be used to guide preventionrelated policy and program decisions in this CTC community.A secondary goal was to share this information with youth development professionals to inform prevention efforts in other communities as well.To facilitate the use of these findings, the study took the form of a utilization-focused evaluation as described by Patton (2008).Throughout each stage of the project, CTC stakeholders were actively engaged in making decisions.Specifically, administrators from the county's Department of Human Services and one member of the county's Board of Commissioners helped identify the research questions, secure data from diverse sources, and provide feedback to the author of this paper throughout the process.As a result, the stakeholders mentioned above were invested in the process and are committed to using the findings to help make policy and program decisions.Together, we developed the following research questions: • What were the dates of administration of CTC evidence-based prevention strategies?
• Were the evidence-based strategies fully or partially implemented?
• Did the level of targeted risk and/or protective factors change following implementation relative to members of a normative database?
• Did the levels of substance use and delinquent behaviors decline following implementation?

Study Design
Implementation data were drawn from several places.First, the CTC stakeholders provided a list of agencies responsible for implementing evidence-based programs within the county.A team of undergraduate students and the author of this paper contacted 18 community agencies and school district offices to obtain information about implementation dates and level of implementation (full program, or specific components).Missing information was filled in from the records of the Prevention Research Center, based at the Pennsylvania State University.Basic CTC planning data, including targeted risk and/or protective factors, were gathered from the official records of the Pennsylvania Commission on Crime and Delinquency (PCCD).
Outcome data were drawn from the Pennsylvania Youth Survey, a paper and pencil survey administered to a representative sample of more than 30,000 Pennsylvania adolescents every two years.We drew data for a single county from the 2001 (n=3498), 2003 (n=4306), 2005 (n=4130), and 2007 (n=3093) waves of data collection.All of the data presented were drawn from technical reports that were aggregated at the county level and were specific to the county being studied.These reports were compiled by consultants who were under contract to the PCCD.The Pennsylvania Youth Survey is based on the Communities That Care Youth Survey, which is a well-validated index that measures risk factors, protective factors, and behavioral outcomes such as delinquency and substance use (Arthur et al., 2002).

Findings
Program Implementation Implementation records revealed that 10 prevention programs were implemented as a result of CTC between the years of 1999 and 2008 (see Figure 1).Most of these programs were fully implemented, although as noted in the figure, three programs were modified to fit within specific time constraints.Programs were implemented in a variety of settings, including schools, communities, and families.A review of PCCD records revealed the CTC system in this county was designed to address the five risk factors of early and persistent antisocial behavior, poor family supervision, poor family discipline, favorable attitudes toward antisocial behavior, and favorable attitudes toward drug use.Early and persistent antisocial behavior refers to experimentation with substances and delinquent behaviors at a young age.Poor family supervision refers to how aware parents are of their adolescents' behaviors and whereabouts.Poor family discipline refers to problems with rules and consequences for behaviors.Adolescents with favorable attitudes toward antisocial behavior and drug use are those who do not believe that such behaviors are wrong.While some Communities That Care communities also focus on protective factors in addition to risks, the county described in this study chose not to do so.

Findings Related to Risk Factors
Tables 1 and 2 include scores for each of the risk factors across multiple years of data collection.Each of these tables includes scores that were measured against a normative database.The normative sample, which includes data from nearly 300,000 participants, was drawn from students from several states whose characteristics are representative of adolescents in the general United States population.Data were coded such that a score of 50 represents the mean for the normative population.Scores above 50 indicate rates that were above the mean, and scores below 50 indicate rates below the mean.Table 1 includes normative scores when all grades levels were combined.These data indicate a general downward trend in risk factors across the years, with changes most evident in attitudes toward problematic behaviors.2001, 2003, and 2005.As the reader interprets these scores, it is important to note that each grade-level was compared to the normative averages for all grade levels combined (e.g., 6 th graders in the county of interest were compared to all grades combined in the normative sample).As a result, younger adolescents tended to have scores below normative averages, while older adolescents tended to have scores above normative averages.In light of the way data were coded, we focused on changes in scores across the years within each grade level, as comparisons across grade levels are not meaningful.Together the data indicate a downward trend in most risk factors across the years.Such trends were most notable among the 10 th and 12 th grade participants.Findings Related to Outcomes The findings for the substance use and delinquency outcomes showed a similar pattern (see Tables 3 and 4).Specifically, rates of most of the outcomes decreased over time.Such declines were not as evident in the 12 th grade students, suggesting delayed onset rather than complete prevention.Since complete prevention is typically not considered an achievable goal, many prevention scholars focus on delayed onset as an indicator of success (Jenson & Fraser, 2006;Olson, 2010).Data for both substance use and delinquency are based on straight percentages rather than comparisons to a normative sample.Because these data do not incorporate a reference group, we cannot be confident that they are due to CTC.Indeed, it is possible they reflect a general downward trend in such behaviors, which has been observed in other studies, such as the Monitoring the Future project (Johnston, O'Malley, Bachman, & Schulenberg, 2009).

Discussion
Together, these data provide preliminary evidence that the CTC system had a positive impact on adolescents.Indeed, seven prevention programs were fully implemented as a result of the needs assessment data collected through the CTC process, and three were partially implemented.Further, data suggest a general decline in the targeted risk factors; and also a decline in the outcome behaviors.The largest changes for the latter occurred in the 6 th and 8 th grades.Such findings are promising as many prevention scholars agree that delaying the onset of behaviors should be a major goal for prevention initiatives (Jenson & Fraser, 2006).
Although we cannot completely rule out alternate explanations for the findings, we have some confidence that the trends related to risk factors are indeed due to CTC, since these data reflect responses from adolescents living in the CTC county as referenced against a normative sample of adolescents from other parts of the country.Specifically, the data indicate adolescents living in the CTC county have shown more improvement over time on the variables of interest as compared to the reference group.This is evident in the way the scores move farther away from the 50 th percentile over time.Such findings are consistent with the statewide evaluation of CTC in the commonwealth of Pennsylvania (Feinberg et al., 2005).

Some Limitations of the Study
The data on outcomes should be considered preliminary.As noted above, these data do show declines across the years, but they are percentages that have not been referenced against a normative sample.They may simply reflect downward trends on rates of substance use and delinquency that have been observed in national surveys of adolescents in recent years (Johnston et al., 2009).In the future, we hope to identify a local reference group we can use to statistically compare the outcomes in this sample to outcomes in a non-CTC community.Finding a suitable comparison community in Pennsylvania is extremely difficult, however, given the significant emphasis on prevention statewide (i.e., the majority of communities have some kind of prevention programming in place).
Another, more general, limitation of the current study is that all data were self-report.Such methods introduce the possibility of bias from lying, forgetfulness, or other inaccuracies in adolescents' own reports of their behaviors.In light of these limitations, we consider this evaluation to represent a first step in assessing the effectiveness of the CTC system in the target county.Despite the limitations, the current findings are consistent with evaluations of CTC that have demonstrated positive effects on both risk factors and other outcomes (Feinberg et al., 2005;Hawkins et al., 2008).

Implications for Practitioners
The current findings suggest that CTC may be a good choice for applied youth development professionals interested in implementing prevention strategies within their local communities.
The system is very comprehensive and provides structure to community efforts to mobilize a variety of groups to work on each aspect of prevention planning, from coalition formation, through program selection and implementation, to interpretation of evaluation findings.The system has a strong theoretical and empirical base, yet can be tailored to the unique characteristics and needs of individual communities.As a result, positive effects in this and other outcome evaluations are not surprising (Feinberg et al., 2005;Hawkins et al., 2008).
CTC may be appealing to local community members because it is readily available.Several years ago, the system was purchased from a private publishing company by the Substance Use and Mental Health Services Administration (SAMHSA).Since this change, all CTC materials are now available free of charge.Resources including training and implementation manuals, needs assessment materials, and a variety of other relevant materials are available through the website of the Social Development Research Group (SDRG), which is affiliated with the School of Social Work at the University of Washington (http://www.sdrg.org/ctcresource/).One potential downside of the program transfer from the publishing company is that technical support is not as readily available as in the past, and the materials are not regularly updated.However, the savings in cost might be appealing to local communities, and contact information is available for communities that need support when implementing specific components of the system.
While most agree that the field of prevention science has room to grow (Olson, 2010), outcomes such as those presented here suggest that current prevention strategies have improved dramatically over early efforts.CTC, in particular, appears to be a positive resource currently available to help applied youth development professionals and others interested in prevention programs work through the long and complex process of implementing strategies likely to be effective.By implementing theory-based initiatives such as CTC, we may be in a strong position to prevent problems and foster positive development among adolescents in our local communities.
Figure 1Implementation dates of evidence-based programs*

Table 1
Normalized risk factor scores averaged across all grade levels

Table 2
includes normative scores by grade level.Unfortunately, due to changes in data coding in 2007, these data are only available for

Table 2
Normalized risk factor scores by grade level

Table 4
Outcome variable percentages by grade level (30-day prevalence rates)