Building a Launchpad for Youth Impact and Organizational Change

A recent report, 4-H Critical Indicators of Youth Development Outcomes for Mission Mandates, outlines a nationwide evaluation of youth program quality and impact of three new programming initiatives. The plan is presented as a model for youth development impact and organizational change. Discussion focuses on the three components of the plan, including evaluation context, framework for assessing program quality and outcomes, and implementation issues critical to successful evaluation.


Introduction
Community-based youth programs demonstrate their social value and funding merit with evidence for program quality and impact (Eccles & Gootman, 2002).Evaluation of quality and impact is a daunting task for local programs, but cumulating evidence across networked youth organizations is even more so.Nevertheless, the prospect of improving program quality and impact and increasing funding support led National 4-H Headquarters to embrace that challenge.4-H developed a long-term strategy for multi-state collaboration that focuses on National Mission Mandates (MM) in science, engineering, and technology (SET), citizenship, and healthy living.
With funding from National 4-H Council, a white paper was commissioned to review literature and recommend impact indicators and procedures for multi-state data collection in mandate areas.The report is part of a broader federal effort to support state and local 4-H professionals who will organize, implement, and evaluate MM programs across diverse populations and settings.
The purpose of this article is to summarize the rubrics and recommendations of that report (Silliman, 2007) which may be relevant across a variety of networked youth organizations.Specifically, this discussion focuses on the evaluation context, framework for selecting and identifying indicators, and recommendations for implementing a multi-state evaluation.

Evaluation Context
Evaluation of nationwide impacts and program quality is part of a larger process of program development and organizational change (Boone, Jones, & Safrit, 2002).Program development begins with critical assessment of organizational and environmental assets and challenges.In this regard, 4-H mission mandates were congruent with the mission of the larger organization, the Cooperative State Research, Education, and Extension Service (CSREES, 2007).Mandate areas were also consistent with 4-H historical programming priorities (National 4-H Headquarters, 2007) and current goals for increasing enrollment in cutting-edge topics.
Organizational assets for implementing and evaluating mandate programs include a dynamic and experienced network of partners at the federal (CSREES), state (land-grant universitybased Extension services), and local (county Extension/4-H offices) levels.The Extension system of community education enables a nationwide reach with state and county-level programming flexibility.National 4-H Headquarters, with the support of the non-profit National 4-H Council, have demonstrated capacity to convene, implement, and evaluate special initiatives including Children, Youth, and Families at Risk (CYFAR), Expanded Foods and Nutrition Extension Program for Youth (EFNEP-Y), 4-H Afterschool, EYSC, and Military Family Programs (National 4-H Headquarters, 2007).CSREES planning and accountability protocols (CSREES, 2007) create a common language and procedure for reporting outcomes of all programs.Additional assets include formal and informal communication and support systems (professional contacts, conferences, committees, training and programming collaboratives) and increasing professional interest in improving evaluation (NAE4-HA, 2006).
Organizational challenges in 4-H include a limited capacity for local program evaluation and limited resources (e.g., planning, training, and management systems, funding, practical instruments) to build capacity.Moreover, state and local autonomy in selecting priority issues and adapting program models fosters program diversity that is not easily assessed by uniform indicators or measures.Although the federal partner has limited authority and few resources to mandate participation (in spite of the connotation of "mandate"), stakeholder "push" for accountability and leadership "pull" toward program improvement creates a climate for collaboration, hence a need for indicators and procedures to guide collective efforts.

Social/Economic Context
State and national environmental scans as well as reviews of research on youth issues confirmed the significance of mission mandate themes.Health, citizenship, and SET problems increase personal, social, and economic risks while increased assets may benefit individuals and communities exponentially.In each field, community-based prevention and education, reinforced early and often, can foster healthy, engaged, prosperous communities.(see Silliman, 2007 for details).

Knowledge Base
Research and practice provides a broad consensus on developmental risks and assets and youth program quality traits (Eccles & Gootman, 2002;SAMHSA, 2007), as well as key outcome indicators in science (Horton, Gogolski, & Warkenton, 2006), citizenship (Brockman, Tepper, & Russell, 2005;Michelson, Zaff, & Hair, 2002;Roebuck, Tepper, & MacNeil, 2005), and health promotion (Centers for Disease Control and Prevention, 2006;Healthy People 2010, 2007).However, social indicators of youth well-being tend to be • incidental (one-time or short-term vs ongoing), • inconsistent (varying in description or measurement), • biased toward inadequacies (problem-focused rather than asset-focused), and • incomplete (focused on one or a few traits) (Moore, Lippman, & Brown, 2004).The process also provides flexibility for mandate-related teams to refine the process as their goals emerge, resources increase, and networks expand.

Evaluation Framework
Based on current research, summarized below, the Critical Indicators evaluation framework targets improvement in content indicators as well as life skills such as communication, goal setting, problem solving, self-efficacy, and teamwork (see Silliman, 2007 for details) as primary evidence for program effectiveness.Based on youth development practice (Kress, 2004) and research (Eccles & Gootman, 2002) the report recommends evaluation context indicators as evidence that content outcomes likely resulted from planned programs.Given the complexity of the project, specifying criteria was an important prerequisite to selecting indicators.
Evidence for program effectiveness must specify audience and setting as well as treatment intensity, frequency, and duration (Chaput, Little, & Weiss, 2004;SAMHSA, 2007).Although 4-H programs are implemented differently across a variety of settings, audience and program descriptors were recommended to facilitate grouping and comparison of programs.Guidelines for implementing multi-state program evaluations were also emphasized to improve the quality and consistency of the evaluation process.Finally, the report suggested promising opportunities for more in-depth research in each mandate area.
Each component of the framework, including criteria, indicators, descriptive data, implementation, and research topics, are discussed below.

Criteria for Selecting Indicators
Evaluation of program outcomes and quality begins with selection of appropriate indicators.In their discussion of criteria for selection of indicators of youth well-being, Moore, Lippman, & Brown (2004) pointed to priorities that are also useful for developing outcome indicators for 4-H mission mandate programs.In this view, indicators should: 1) represent all domains (e.g., physical, cognitive, social development); 2) describe social context (e.g., community or delivery system); 3) be sensitive to developmental stages and; 4) linkages between stages; 5) include positive development as well as risk factors; 6) address factors sensitive to short-, medium-, and long-term influences.
Reviews of research identified the domains and factors appropriate to each mandate-area in the short-, mid-, and long-term time frames.Numbers of youth increasing in outcomes is incorporated, consistent with the 4-H goal of increasing enrollment.Developmentally-sensitive indicators are left to mandate planning teams as they target age-specific audiences.
Review of the evaluation context suggested additional criteria, including: 1) consistency across mission mandate areas; 2) consistency with national indices such as science education standards, health surveillance systems; 3) consistency between routine monitoring (e.g., USDA reporting), evaluation, and research levels; and 4) understandable to citizens, practitioners, researchers, and policy-makers.
Uniformity of indicators (items 7-9) would facilitate planning, implementation, and evaluation across the CSREES system, comparisons with similar programs in the field, and increasing efficiency of effort for 4-H staff in completing CSREES reports and evaluating mandate-related programs.The final criterion will enhance implementation and marketing of program evaluation results.
Criteria for selection of program quality indicators include practices that: 1) reduce risk behaviors; and 2) promote positive youth development; 3) are culturally sensitive (Eccles & Gootman, 2002) and 4) match the developmental and risk level of the target audience (SAMHSA, 2007).
Model practices are those associated with programs shown to produce positive outcomes in multiple studies and settings (SAMSHA, 2007).

Content Evaluation
Programs demonstrate worth not simply by conducting activities or serving large audiences, but by facilitating meaningful change in the lives of participants.Specifying program outcomes establishes the goals and describes the results of educational programs.In the CSREES evaluation logic model, outcome indicators specify measurable characteristics that can be tracked across time, participants, and location (Bennett & Rockwell, 1995;CSREES, 2007) 2000), citizenship and volunteerism (Brockman, Tepper, & Russell, 2005;Zaff et al., 2002), and health promotion (Centers for Disease Control, 2007; Kahan & Goodstadt, 2001).
The Critical Indicators framework recommends youth-friendly measures of program outcomes and quality.Authentic assessment that informs, engages, and encourages young people contrasts high-stakes testing environments in which evaluation is often impersonal and irrelevant for youth.Even adaptations of knowledge test formats such as adding a debriefing dialogue may help leaders better understand youth needs and affirm youth voice, as well as evaluate program outcomes.Multiple perspectives, including evaluations of youth, leaders, parents, and others also create a fuller picture of the quality or impact of a youth program.Integration of evaluation methods with educational and relational experiences should provide more reliable reports as well.
The Critical Indicators framework provides a broad rubric for short-, mid-, and long-term outcomes that each mandate leadership team can refine to fit the type and degree of change expected with each program or curricula.Organizational goals for increasing enrollment through mandate-related programs are also incorporated into indicators statements.Although numbers of participants represents output, rather than outcome-level data, increasing numbers of youth who increase specific skills and behaviors represents a priority goal within the organization.Benchmarks on both enrollment and achievement provide valuable feedback on the success of the initiative.

Demographics and Descriptors
The diversity of people, programs, and settings included in 4-H mission mandate programs commends documentation.Mandate leadership teams may choose to pre-screen participating programs to conduct a more uniform evaluation of a curriculum or program.By contrast, mandate teams may admit any 4-H group that identifies with their theme.In both cases, demographic and descriptive information will aid evaluators in determining the effect of each type of program on each type of participant in each type of setting.Thus, Critical Indicators recommended gathering information about the audience.Participant data might include demographics (age/grade, gender, race, place of residence, disability status, primary language) and descriptive data (sessions attended, prior knowledge, 4-H experience).Program data must describe the intensity or frequency of training, duration, or length of exposure, depth, or mastery level, and breadth, or diversity of components (Chaput, Little, & Weiss, 2004).Other valuable program data may include information about program format or delivery system (e.g., clubs, afterschool, special interest, camps), traits of the curriculum and/or program, practices, including instruction, experimentation, discussion, or demonstration, or identification as a model program (Elliott, 1999;SAMHSA, 2007;U.S. Department of Education, 2003).
Descriptions of leader traits such as age, gender, race, experience, and role may also prove useful in evaluating program effectiveness.Descriptions of the setting or environment, including available supports and opportunities (Gambone, Klem, & Connell, 2002) for program goals in families, schools, youth organizations, and communities may contribute to understanding how programs impact participants.

Research Opportunities
Mission mandate programs may provide valuable opportunities for systematic and in-depth research beyond the scope of routine evaluation.The Critical Indicators framework recommends several research topics for each mandate area (see Appendix B).Investigations in both basic and applied research may fill gaps in existing research on science, civics, and health for youth.Research on youth programs may also strengthen links with higher education partners and funding agencies.

Implementation
Implementation represents the final and pivotal stage of the Critical Indicators project.Mandate leadership teams of 4-H professionals will review the report, available curricula, and training needs, then develop plans for engaging community-level programs.The SET team, now mobilized, is reviewing curriculum capacity and conducting an online survey with local 4-H staff on their readiness for SET programming.Similar profiles of program capacity in each area would help target training and resource needs and benchmark program growth.Building on these profiles, leadership teams can deploy packages of programming and evaluation resources that establish and expand upon best practices in the field.
Using the program incubation process developed by Cornell Extension (Hertzog, 2006), mandate program partners should initially focus on program quality and fidelity, recognizing that good practice represents the strongest correlate of and foundation for program impact.
Programs at the development phase can begin using pre-and post-tests, targeting short-and mid-term indicators and descriptors.Mature programs, with stable procedures and consistently positive results can conduct in-depth evaluation using quasi-experimental and experimental designs.At each phase, a tool-kit of programming and measurement tools, juried curricula and support resources, program and data management procedures, recruitment and marketing resources will improve community-level programming and evaluation results.Deployment of these systems will provide many opportunities for research on program effectiveness and longterm impact.

Conclusions
The Critical Indicators report prepared to support the 4-H national Mission Mandates addresses the context, framework, and implementation steps needed to evaluate multi-state programming initiatives.As the foregoing review suggests, youth development organizations can realize significant benefits from implementing such a plan, including: 1) greater focus in program planning, as collaborators "begin with [a shared] end in mind;" 2) greater richness in program development, as a variety of strategies are logically linked to a shared outcome; 3) greater continuity in curriculum and training as they are aimed toward shared 4) opportunities to compare delivery methods for specific audiences; and 5) more clear and powerful evidence for program impact as all efforts contribute to a single, larger impact statement.
Such a plan also challenges youth organizations to assess and invest in programming and evaluation capacity through: 1) upgrading existing systems for program planning, implementation, documentation, and reporting in ways that enhance both innovation and youth worker self-efficacy; 2) engaging professionals and volunteers in program development and implementation and enhancing their experience with training, networking, and support experiences; 3) building organizational awareness of and commitment to evaluation as a strategy for clarifying mission, communicating with clients and partners, improving programs as well as documenting their results, tracking processes that increase program and organizational effectiveness, and communicating with all stakeholders.
The Critical Indicators framework provides a first step toward realizing these benefits for 4-H and a model that may be useful to similar organizations with nationwide reach.Future efforts by 4-H or other youth organizations should address needs for the following: 1) establishing more consistent indicators for youth development in general, as well as specific areas of programming (e.g., health, citizenship, SET), program traits and quality, critical participant characteristics, and descriptors of processes for building capacity, evaluating programs, and using evaluation data; 2) training and empowerment of youth development professionals and volunteers to engage in collaborative efforts as well as develop indicators and methods to support unique programs of their own; 3) developing organizational capacity to lead and support professionals in collaborative efforts; and 4) enhanced scholarship to develop and improve the discipline of youth development evaluation especially for community-based programs (vs.research model programs).
The Critical Indicators report provides a framework for evaluating 4-H national Mission Mandates and a window to the myriad of challenges and opportunities facing youth professionals, programs, and organizations.As these challenges are documented, discussed, and solutions determined, youth program effectiveness and subsequent support promises to increase.• Goal-setting Number of individuals increasing skills in setting and completing goals on a project (short-term), then applying skills to a self-directed or community leadership project or teaching skills to others (long-term) • Critical thinking, Problem solving, Inquiry skills Number of individuals increasing skills in reasoning on a project (short-term), then applying skills to a self-directed or group leadership project or teaching skills to others (long-term) • Self-efficacy Number of individuals increasing self-efficacy (confidence to perform), then attribute their confidence in a career or community service task to 4-H (long-term) • Teamwork Number of individuals increasing skills as a team member (short-term), then applying skills as a team leader (long-term) measures of progress • Number of individuals increasing participation in science and technology-specific clubs • Number of individuals increasing knowledge and/or skills in content and careers (across subject areas ranging from animal science to technology) • Number of individuals increasing positive attitude and/or aspirations about SET learning and careers • Number of individuals increasing science process skills, such as mastery of basic skills (observation, comparison, hypothesis), use of the scientific method, or systematic problem solving Mid-term indicators reflecting application of knowledge and practice of skills • Number of individuals demonstrating improved behavior in science learning, such as effective participation in school classes, independent study, career exploration, or volunteer experiences • Number of individuals applying science process skills, including incorporation of science learning in community service, enrollment in SET-related post-secondary education, and/or entrepreneurship/career success Long-term indicators of personal or social change • Number of individuals who enroll in SET-related post-secondary education, enter SET-related career fields, demonstrate entrepreneurship/career success in a SET field, or attribute 4-H SET involvement to success in education or career • Number of individuals who sustain 4-H SET involvement as volunteer leaders, community or corporate SET decision-makers, and/or adult sponsors of 4-H SET activities Citizenship and Volunteerism Short-term measures of progress • Number of hours in civic involvement • Number of hours in community service • Number of individuals increasing knowledge and/or skills in civic education, including the elections process • Number of individuals gaining knowledge and/or skills related to volunteerism • Number of individuals reporting positive attitude change and/or aspiration related to volunteering for civic activities, community service, and/or philanthropy • Number of individuals indicating knowledge and/or skills learned related to leadership • Number of individuals reporting positive attitude change and/or aspirations related to leadership • Number of individuals indicating knowledge and/or skills gained related to effective youthadult partnerships • Number of individuals indicating positive attitude change and/or aspirations related to participate in youth-adult partnerships Mid-term indicators reflecting application of knowledge and practice of skills Gaps remain in developmental and program research findings in each mandate area.For instance, "typical" patterns of health and nutrition are changing, many strategies for engaging youth in science or citizenship remain untested, and program outcomes may vary across Horton, Gogolski, & Warkenton (2006)00) best determined, 2007search and practice in each mandate area.In the absence of subject-specific data, broader youth development research and/or theory must be applied to select appropriate indicators.Research-based model programs typically specify the degree and timing of change.Without research-based norms, outcome time-frames are relative descriptions; that is, the results of immediate or sustained learning and practice.Within this framework short-term refers to results of one week (intensive) to one year (less intensive) programs.Mid-term describes results of roughly three-month (sustained practice) to three year (extended practice).Longterm outcomes anticipate attitudes and behaviors that extend into late adolescence and young adulthood.Obviously, participant age and capacities, program intensity and duration, environmental supports and opportunities significantly influence the scope of outcomes in a given time frame.These issues are addressed further under Demographics and Descriptors.Outcome indicators for each mandate area were selected from best practice research.National science education standards(National Research Council, 2007, 2000)recommend outcomes based on research with youth, grades K-16.Horton, Gogolski, & Warkenton (2006)identified key outcomes in these standards as anchors (discipline-base knowledge) and abilities (processes such as observation and problem solving).Juried 4-H curricula already reflect these outcomes.SET indicators (see Appendix A) focus on acquisition of competencies and aspirations for learning in the short-term, mastery and flexible application in the mid-term, and expanded engagement through career and volunteer activity in the long-term.Health indicators, including mental and physical health and safety, are drawn from research and surveillance data of the U.S. Departments of Agriculture and Health and Human Services (Centers for DiseaseControl and Prevention, 2006, 2005; Healthy People 2010, 2007; U.S. Department ofAgriculture, 2006Agriculture,  , 2005; U.S. Surgeon General, 2001).Short-term indicators (see Appendix A) focus on gaining knowledge and skills for healthy living, mid-term indicators emphasize using knowledge and skills to practice more healthy and safe habits, and long-term outcomes target healthy lifestyles into teen and young adult years.Life skills represent key competencies across content areas, long valued as outcomes of 4-H and other informal education programs(Eccles & Gootman, 2002).Life skills deemed most relevant to target programs include communication (oral, written, interpersonal), goal setting, problem solving (scientific and interpersonal), self-efficacy, and teamwork.Although life skills are frequently cited in youth outcomes research(Elliott, 1999; SAMHSA, 2007; U.S. Department  of Education, 2003), no developmental rubric could be located for monitoring growth and application of these skills over time.Thus, generic indicators, rather than short-and long-term indicators, were used to target outcomes in the Life Skills arena.Evaluation Program quality is the first evidence of worth and the final evidence for program impact.Programs in a formative or developmental stage typically emphasize program quality or fidelity as evidence of progress toward targeted outcomes.However, evidence for program quality during summative evaluation strengthens the case that outcomes resulted from a planned program.A National Research Council review of program qualities that promote positive youth development(Eccles & Gootman, 2002)identifies the following criteria: Many of these qualities are reflected in research-based model programs(Elliott, 1999; SAMHSA,  2007; U.S. Department of Education, 2003)as well as 4-H Essential Elements of 4-H (University ofArizona, 2006), and High Scope Youth Program Quality Assessment (2004).Best practice reviews are also available in disciplines such as science inquiry(National Research Council, . Following this model, short-term iMichelson, et al., 2002;Roebuck, Brockman, & Tepper, 2005)or aspirations (KASA).Over time, application of short-term KASA changes should be evident in behavior changes.Over an extended time, individual and group-level behavior changes may result in long-term social, economic, environmental, or cultural change.Citizenship indicators were drawn from reviews of research(Brockman, Tepper, & Russell,  2005;Michelson, et al., 2002;Roebuck, Brockman, & Tepper, 2005)and recommendations on civic awareness and engagement(Civic Indicators Working Group, 2007; North Carolina Civic  Consortium, 2003), and volunteer service and leadership (America's Promise, 2007; Corporation forNational and Community Service, 2007).Citizenship indicators (see Appendix A) focus on familiarity and activity in the short-term, shift to expanded activity and leadership initiative in the mid-term, and describe sustained involvement in adulthood.
• Number of individuals engaged in the political/governance process, including registering to vote, attending meetings or governmental or civic organizations, presenting on civic boards, writing to elected or civil service officials, meeting with legislators, seeking office in a club or school, publishing letter(s) to the editor, or participating in the election process • Number of individuals who increase volunteer responsibilities, including leading a community service project • Number of individuals engaged in youth-adult partnerships, such as serving on a policymaking and/or advocacy board Long-term indicators of personal or social change • Number of individuals who enroll in a public service-related post-secondary education discipline, enter public-service-related career fields, demonstrate career success in a public service field, or attribute 4-H citizenship involvement to success in education or career • Number of individuals who continue civic engagement as adults, including voting, working in the elections process, community or corporate service, and youth-adult partnership boards • Number of individuals who serve as volunteers in youth civic engagement as leaders, community or corporate advocates for citizenship, and/or adult sponsors of citizenship activities Number of individuals increasing knowledge of and/or skills for selecting healthy foods, including understanding food labels, personal dietary habits, portion sizes, and preparation of foods with reduced fat and/or calories • Number of individuals improving attitudes toward and/or aspirations to improve nutritional habits such as eating healthy foods, decreasing sugar-sweetened beverages • Number of individuals increasing knowledge of and/or skills in physical activity and reducing risk behaviors such as excessive screen time • Number of individuals improving attitudes toward and/or aspirations to improve physical activity habits and reduce risk behaviors • Number of individuals increasing knowledge of and/or skills in practicing healthy habits, including adequate sleep, personal hygiene, dental care • Number of individuals improving attitudes toward and/or aspirations to improve health habits related to sleep, hygiene, dental care • Number of individuals increasing knowledge and/or skills related to safety, including ATV, bike, hunter, and water sports • Number of individuals increasing stress coping skills • Number of individuals increasing refusal skills related to substance abuse and violence • Number of individuals increasing knowledge of personal and community resources for mental health including where to seek help when facing depression, anxiety, suicidal thoughts, violence, addiction, or eating disorders Mid-term indicators reflecting application of knowledge and practice of skills • Number of individuals maintaining positive nutrition habits (identified via short-term indicators), with positive consequences such as weight control, reduction in health-related problems • Number of individuals maintaining physical activity habits (identified via short-term indicators), with positive consequences as noted above • Number of individuals maintaining safety practices (identified via short-term indicators) and decreasing injuries in targeted activities • Number of individuals maintaining positive health habits (identified via short-term indicators) and decreasing risk behaviors Long-term indicators of personal or social change • Number of individuals who enroll in Health-related post-secondary education, enter Healthrelated career fields, demonstrate entrepreneurship/career success in a Health field, or attribute 4-H Healthy Lifestyles involvement to success in education or career • Number of individuals who maintain healthy lifestyles into adulthood, including good nutrition, weight control, regular exercise, good mental health habits, safety habits, and risk avoidance • Number of individuals who sustain involvement with youth health programs as volunteer leaders, community or corporate health decision-makers, and/or adult sponsors of 4-H healthy lifestyles activities Life Skills Indicators across all mission mandates • Communication Number of individuals increasing skills in public speaking (short-term), then applying public speaking skills in another setting (long-term) Number of individuals increasing skills in record-keeping (short-term), then applying writing skills to a more complex task (long-term) Number of individuals increasing skills in interpersonal communication (short-term), then demonstrating competence in communicating as a leader (long-term)