Evaluating Youth Programs : An Overview of Practitioner Led Evaluations

Youth programs are prime contexts for the intentional facilitation of positive development. However, not all youth programs achieve positive outcomes equally. In order to promote the identification and dissemination of the characteristics and processes of effective youth programs, increased focus needs to be given to program evaluation. This article briefly reviews the main tenets of evaluation science in order to provide practitioners a roadmap for conducting their own evaluations. This includes an overview of different types of evaluations and key issues to consider when constructing an evaluation strategy such as targeting outcomes and developing program logic models.


Introduction
Programming is a central component of positive youth development (PYD; Whitlock & Hamilton, 2001, as cited in Small & Memmo, 2004).Youth programs are one of the key contexts in which adolescents can receive the necessary supports, opportunities, and services that they need to develop into fully functioning adults (Furstenberg, Elder, Cook, & Eccles, 2000;Whitlock, 2004).This focus on quality programming and positive outcomes requires a stronger emphasis on program evaluation (Catalano, Berglund, Ryan, Lonczak, & Hawkins, 2002).Quality evaluations can provide important information relative to program improvement, identification and dissemination of best practices, and empirical support for current program efforts.Drawing from the work of Rossi, Lipsey and Freeman (2004) and others, this paper presents an overview of the field of evaluation science and offers suggestions for how practitioners can evaluate their own programs.

Purpose of Evaluations
Evaluations produce information that can be of use to a variety of stakeholders (Rossi, et al., 2004).Practitioners gain information regarding program implementation, outcomes, and insights that can be utilized as part of a comprehensive program improvement model.Program funders can use evaluation findings to make important decisions regarding whether to fund or continue to fund programs.Evaluation findings can be used by policy makers to judge the effectiveness of an initiative and inform future policy decisions.Lastly, potential and current program participants can use evaluation findings to inform their decisions pertaining to participation.

Initial Steps
The foundational steps to conducting an evaluation include identifying the main objectives of the project (e.g., program improvement, accountability, knowledge generation, political purposes or public relations); the questions that need to be answered; and the goals and targeted outcomes of the program (Rossi, et al., 2004).A clear understanding of program objectives is especially important since this knowledge facilitates an assessment of the program's theoretical foundation which in turn helps determine the most appropriate type of evaluation to conduct.
The logic behind why a program is expected to produce targeted outcomes is known as program theory (Rossi, et al., 2004) and it usually exists in one of two forms.If you have a clear understanding of what a program does and why it works your theory is articulated but if you struggle to explain the processes and impact of their program the theory is only implied (Weiss, 1997).In the case of a program with only an implied theory, practitioners need to develop a more articulated theory before attempting to conduct an evaluation (Rossi, et al.).Developing a logic model for the program represents one approach for both identifying the outcomes and articulating theory.This is a topic that has received considerable attention elsewhere and readers are encouraged to refer to existing logic model resources (Baldwin, Caldwell, & Witt, 2005;McLaughlin & Jordan, 1999;Renger & Titcomb, 2002) for further information on this topic.

Types of Evaluation Strategies
Once practitioners have identified/clarified program objectives and theory, they are ready to select an appropriate evaluation strategy.The following sections will outline a number of evaluation strategies as described by Rossi et al. (2004).The strategies build sequentially, with each providing information necessary to the implementation of the next.
Needs assessment.The first type of evaluation strategy is the needs assessment, which involves, as the name implies, assessing the actual need for the program.For example, if the program in question is a teen pregnancy prevention program, a needs assessment would seek to establish the prevalence of pregnant teens and potentially at-risk populations within the program's service area.
Program theory assessment.Program theory assessments involve developing a working, theoretical model of the program.An effective assessment of this nature takes into consideration each separate aspect of the program theory, namely the impact theory, service utilization plan, and organizational plan (Rossi, et al., 2004).The impact theory justifies why the program should produce its targeted outcomes.The service utilization plan describes how the program will provide key services to its targeted population.Lastly, the organizational plan details how available program resources will be organized to accomplish tasks prescribed in the service utilization plan.
Program process assessment.This type of assessment is usually focused on answering questions regarding whether or not services are being provided to the target population and how well the program's service delivery processes match program design expectations (Rossi, et al., 2004).The two main types of program process assessment are continuous program monitoring and implementation evaluations (Rossi, et al.).The first of these, program monitoring, involves the ongoing assessment of key indicators (e.g., number of participants served per week) related to program processes.
The second, implementation evaluations, consists of assessing what is usually referred to as implementation integrity, which is simply the degree to which a program is run as originally planned.Implementation integrity consists of the following components: program adherence, dosage, quality of delivery, and participant satisfaction (Dane & Schneider, 1998).Program adherence measures if the program is implemented as planned; dosage refers to the amount of program services each participant receives; quality of delivery deals with the qualitative nature of program delivery; and participant satisfaction is a subjective measure of how satisfied individuals are with the program's services.Information needs to be gathered in each of these domains in order to effectively assess implementation integrity.
Implementation evaluations allow practitioners to better understand the processes that produce program outcomes (Chen, 1998).They also help guard against attributing outcomes, or lack thereof, to a program when in fact the outcomes were caused by some external, unaccounted for factor (Dobson & Cook, 1980).Such a situation arises due to implementation failure, which signifies that a program did not produce its desired outcomes, not because of a faulty conceptualization and design (i.e., theory failure) but because it was not implemented correctly (Rossi, et al., 2004).
Implementation findings help practitioners to describe the necessary program components and degree of program integrity needed to produce outcomes, thus enabling more successful replications of the intervention (Durlak, 1998).Implementation data also enables practitioners to more accurately determine key components of the program responsible for observed changes (Durlak, 1998).Implementation evaluations can produce applicable information that can potentially improve program performance and effectiveness.Impact assessments.Once the program theory has been articulated and a strategy is developed to assess program implementation, evaluators can turn their focus to measuring outcomes.For this step, the initial development of a logic model will be very useful because targeted outcomes will have already been identified.The focus of impact assessment should be on measuring those outcomes for which the program will have the most immediate impact.For example, while an overarching goal of an after-school program may be to help students enter college, the program's more immediate impact will be better assessed by an assessment of home work completion rates.

Data Collection
After selecting the type of evaluation, an articulated program theory can provide a blue print to identify what type of information should be gathered.Practitioner's knowledge regarding their program(s) logistics and participants' characteristics can inform the data collection strategy.While the temptation to answer as many questions as possible can be strong, practitioners need to balance the desire for information with a clear assessment of how much times participants will actually be willing to spend completing a survey.Decisions related to the mode of data collection are also important.Survey research has traditionally relied on pen and paper questionnaires but the availability of low to no cost online survey methodologies is increasing the popularity of this option.

Interpretation and Implementation of Findings
Although the thought of conducting statistical analyses may cause some trepidation, a wealth of knowledge can be gained through simple descriptive statistics (e.g., mean, standard deviations, etc.).While more complicated analyses can also provide important answers, important and applicable information can be gained from evaluation results without relying on complex statistical procedures.In fact, practitioners may face more difficulties implementing than interpreting their findings.A plan needs to be put in place before the evaluation commences to promote the eventual application of findings.Without such a piece, even the best crafted evaluation's impact will be greatly diluted.

Summary
Although providing youth with opportunities to participate in programs is a worthwhile goal, considerable time, money, and effort can be wasted on ineffective programs.Practitioners should thus be engaged in the evaluation of programs.The guidelines presented in this article should help make this process more accessible and serve as a starting point for future, practitioner driven, evaluations.The healthy development of youth is at stake and this fact alone makes it essential that high quality program evaluations become a hallmark of PYD.The better understanding we have of PYD best practices and processes, the more effective programs can be implemented and more youths' lives can be positively impacted.