Process Evaluation and Continuous Improvement in Community Youth Programs

A method of using process evaluation to provide improvement plans in order to promote community youth programs is described. The core elements of this method include the following: (1) collection and analysis of baseline data, (2) feedback provided to programs describing their strengths and limitations, (3) programs provided with assistance in preparing improvement plans in regard to their baseline data, and (4) follow-up evaluation assessed program changes based on their improvement plans and baseline data. A case study of an inner-city neighborhood youth center is used to demonstrate this method.


Introduction
Numerous evaluations of youth programs founded on youth development principles have been conducted.However, most of these evaluations focused on youth outcomes, and more specifically, on whether a program had a positive influence on youth participants' psychosocial development (Allen, Philliber, Herrling, & Kupermine, 1997;Catalano, Berglund, Ryan, Lonczak, & Hawkins, 2002;Hawkins, Catalano, Kosterman, Abbott, & Hill, 1999;Roth, Brooks-Gunn, Murray, & Foster, 1998).There is a growing sentiment in the youth development field, however, that evaluations must focus more on what, in fact, is going on within the programs and whether programs are fulfilling the objectives of the youth development approach to programming (Catalano et al., 2002;Roth et al;1998).It is not enough to know that a given program works.It is also important to know whether programs that work actually adhere to a youth development model.Thus, "process evaluations" are a necessary part of evaluating and refining the programming offered within youth development programs (Catalano et al., 2002;Gambone, Cao, Lewis-Charp, Sipe, & Lacoe, 2004;Gambone, Klem, & Connell, 2003;National Research Council, 2002).

Process Evaluation
Process evaluation is a form of program evaluation that applies descriptive research methods to compare the program being delivered with the program that was originally intended by planners (Stufflebeam & Shinkfield, 1985).Process evaluations can offer program directors a better understanding of how a program concept has been implemented.Additionally, they can provide insight into the strengths and weaknesses of a program's structure and delivery, and they can enhance the ability of program directors to describe their programs to outside sources.Process evaluations also help to identify which, among various program elements, are most essential for promoting developmental change (Larson, 2000).Furthermore, process evaluation data can play a critical role in improving a program's overall impact.
According to Patton (1994), a process evaluation should provide feedback on the original program and lead to improvements in the implementation and effectiveness of the program.Such modifications and improvements, in turn, should lead to improved program outcomes and impacts over time.
Finally, process evaluations can help funders and policy makers make informed choices about which programs to fund based upon the programs' accountability in offering the program as planned.In short, process evaluations can play an important role in describing the extent to which youth development programming is being implemented and in improving the overall quality of such programs.
The steps involved in using process evaluation to promote continuous program improvement are summarized in the next section.This section is followed by a case study of how the process worked in a Boys and Girls Club.

Process Evaluation Steps
Phase 1: A survey was administered to youth participating in the participating programs.The survey used for this purpose is the Youth Development Assessment Device (YDAD; Sabatelli, Anderson, & Rubinfeld, 2006).The YDAD was designed to assess the "developmental quality" of youth programs from the perspective of the youth.Developmental quality is the extent to which a program provides a set of program components that previous research has found to facilitate positive youth development (Eccles & Gootman, 2002).Specifically, questionnaire items were created to assess the following program attributes: a. the presence of supportive relationships; b. the existence of a physically and emotionally safe environment; c. the existence of programs offering challenging activities, and d. the existence of opportunities for youth to be meaningfully involved with their programs.
Each of these conceptual dimensions is characterized by a constellation of interrelated subdimensions.For example, with respect to safety, a high quality developmental youth program creates an environment that is both physically and emotionally safe.With respect to supportive relationships, a high quality developmental program is staffed by individuals who are knowledgeable of youth and who create opportunities for youth to receive guidance, emotional support, and instrumental or practical support.
The dimensions and conceptual sub-dimensions characterizing programs that provide supports and opportunities promoting youth development are summarized in Table 1 on the following page.
Following administration of the survey, results were shared at a feedback session with youth program staff.
Phase 2: Based on the results of the first survey administration, programs were provided technical assistance to create program improvement plans.A liaison worked directly with youthstaff teams to interpret the results of the first round of data and to identify areas for improvement.The youth programs then had the opportunity to implement their plans, with further technical assistance, over a one-year period.
Phase 3: At the end of the project year, the process evaluation was repeated.Once again, the same youth were asked to fill out the survey questionnaire designed to assess their perceptions of and experiences within the programs.These data were analyzed and reports were distributed to the youth programs during a feedback session.
Phase 4: Comparisons between youth's responses on the process survey between Time 1 and Time 2 were used to assess program changes.The two primary questions addressed were as follows: (1) to what degree did youth report changes in the developmental quality of the program they attended, and (2) were the reported changes consistent with the improvement plan that had been developed by each youth program?
Case Study

Process Evaluation Time One
Staff at the Boys and Girls Club administered the YDAD to youth attending an inner city Boys and Girls Club.This data was then analyzed by the authors.The scores of males and females and younger versus older youth were contrasted in order to maximize the usefulness of the data collected from the youth.Results revealed that females scored higher than males in a number of different areas (see Table 1).Increase males' scores on the practical supports available at the center from 13.5 to 14.5.

Proposed Activities
Have staff members approach youth (one on one) to review their membership status.Provide youth with new and improved membership cards and teach them about the many benefits of the new membership system.Reinforce the idea that the Boys and Girls Club is an important membership service organization to belong to.
Boys and Girls Club staff will hold bimonthly group sessions with youth to address their concerns and seek input on program improvements.
Staff will build greater rapport with disengaged youth to enhance their sense of safety and willingness to approach a staff person with personal issues or problems.

Table 1
Average Scores on the Process Indicators: Contrasting Males and Females and Contrasting Younger Youth and Older Youth (n=99) Staff and youth will receive training in Advancing Youth Development practices, Girls and Boys Club of USA Principles of Youth Work, Psychology for Kids, What Teens Need to Succeed, and Conflict Resolution skills building materials.