Tracking Evaluation Capacity with Youth Professionals

A pilot program mentoring youth professionals through “learning-by-doing” projects yielded consistent increases in evaluation knowledge and skills over three years. Self-assessed skill improvements were greatest for preparatory processes (planning, focusing, design, selecting methods) and reporting competencies that are more often emphasized in organizational evaluation requirements. Smaller increases in data collection and analysis skills were also perceived by participating youth professionals. Focus groups with each of six evaluation “learning circle” groups revealed benefits of participation in the evaluation “learning circle,” as well as needs for evaluation training and tools, and challenges faced within the organizational culture.


Introduction
This paper describes the context and process for developing learning circles of communitybased youth professionals to build program evaluation capacity.Improvements in evaluation skills among learning circle members and perceptions of goals, activities, and benefits of learning circle teams are presented.Discussion focuses on implications for practice, research, and policy.
Evaluation skills represent a critical competency for Cooperative Extension's 4-H Program and other youth-serving organizations to address the dual challenges of program accountability and quality improvement (National Association of Extension 4-H Agents, 2006;National Professional Development Task Force, 2004).Capacity-building is a priority for these organizations (Bialeschki & Conn, 2011;Rennekamp & Engle, 2008;Taylor-Powell & Boyd, 2008).Several professional organizations identify competencies considered essential for evaluation practitioners (Archer, Bruns, & Heaney, 2007;Arnold, et al., 2008;OPEN Initiative, 2007).
Although routine reporting of outcomes necessitates a working knowledge or practical awareness of evaluation concepts and procedures, program leadership for youth programs typically requires functional knowledge or mastery of theory and practice in both programming and evaluation (Arnold, et al., 2008).For most community-based professionals, limited training, support, and ongoing job demands make it difficult to achieve functional capacity.
Learning circles which partner a capable, adaptable mentor with highly-engaged professionals and focus on project-based collaboration are identified as both attractive and effective means to strengthen skills and capacities among Extension professionals (Baughman, et al., 2010).This type of small group "learning by doing" or project-based mentoring has been shown effective in building evaluation capacity, particularly when combined with structured opportunities to learn the language and logic of evaluation (Arnold, 2006;King & Volkov, 2005).
According to Brooks (2010), a combination of face-to-face contacts and web-based informational and interactive tools are effective elements in the successful facilitation of university faculty development.Further, Schlager, Fusco, and Shank (2002) report that webbased learning communities are effective for supporting and training new teachers, particularly in "learning the ropes" in their first job.Several training platforms are available for learning about program evaluation online (Archer, Bruns, & Heaney, 2007;Centers for Disease Control, 2011;CYFAR, 2011), with some providing evidence of effectiveness (Fleming & Easton, 2009;Kreis & Wilke, 2009).

Development of NC 4-H Evaluation Learning Circles (E-LC)
Building on these insights, North Carolina 4-H initiated evaluation learning circle teams in January of 2009.Agents with 5-15 years' experience were invited to share in monthly online training sessions and to join self-directed, collaborative, project-based teams mentored by a campus-based specialist.Participants received no incentives other than the opportunity to learn and work together on a professional development and program accountability priority.Groups met online for instruction and discussion at first, but quickly made their own arrangements to balance online and face-to-face meetings by forming regional teams.
In Year 1, 12 staff (6 teams) participated in a monthly webinar on evaluation skills and developed evaluation plans, with one team completing a program evaluation.Nine staff continued in the network in Year 2, joined by 6 peers (6 teams).Teams became more autonomous, with the specialist/mentor acting as consultant rather than as convener.Webbased training was available through the eXtension Evaluation Community of Practice (eXtension Evaluation Community of Practice, 2011), but team members more consistently participated in online or face-to-face mentoring sessions with the specialist every 4-6 weeks.All webinars and online meetings were conducted using Elluminate!Live software.Four teams completed program evaluations and two others identified improvements needed to complete a plan.At the beginning of Year 3, 13 staff continued, joined by 6 peers.Six groups developed logic models, evaluation plans and instruments.Five groups implemented evaluations for programs including 4-H camp and club impact, presentation skills, goal-setting/achievement skills, science fair, youth leadership and service.

Research and Practice Questions
This study sought to determine whether E-LC members were gaining evaluation skills as well as to discover their perceptions of the context, activities, and benefits in E-LC teams.

Sample
Twenty-one of 24 youth professionals who participated in Evaluation Learning Circle (E-LC) teams during its first three years completed an Evaluation Skills Self Assessment (ESSA) pre-test.Fifteen completed an ESSA one year post-test (9 after Year 1, 6 after Year 2).Seven completed the ESSA post-test after one and -two years.Attrition in Years 1 and 2 resulted from medical leave (N = 2) or job changes (N = 3).Six new participants joined or formed learning circles at the beginning of Year 3. Participants ranged in age from 27 to 50 years, 2-27 years of experience.Most participants (N = 21) were Caucasian, 3 were African-American.Fourteen were located in rural communities, 2 in urban, and 8 in mixed rural-urban counties.When they joined an E-LC team, 11 held bachelor's degrees, 8 master's degrees, and 2 doctoral degrees.Of those who continued to Year 3, 5 of the Year 1 cohort held masters or doctoral degrees, and 2 completed an advanced degree during the year.Of those in the Year 2 cohort, only 2 held an advanced degree, although 6 completed their advanced degree during that year.

Instruments, Procedures, and Analysis
Participants completed an Evaluation Skills Self-Assessment (ESSA) (Arnold, et al., 2008) during each year of their participation in the E-LC.This tool, presenting 41 items on knowledge and skills in seven phases of evaluation, was developed by a team of evaluation specialists to target needs for training and document skill growth.Participants rated themselves from "Know Nothing" (0) to "Understand the Basic Concept" (1) to "Can Implement Concept with Assistance" (2) to "Can Implement the Concept Independently" (3).Participants completed the ESSA on joining E-LC teams, then annually thereafter.Data analysis, using SPSS-PC for Mac focused primarily on descriptive statistics given the small convenience sample and attrition within annual samples.Friedman non-parametric t-tests with paired samples were conducted on pre/post scores to track evidence for skill change.
Online focus groups were conducted at the end of Year 2 by a professional evaluator.Members from each E-LC team were asked about prior evaluation experience, goals and expectations for participation with the E-LC, evaluation training and resource needs, perceived benefits of participating in the E-LC, and the evaluation challenges of NC's 4-H agents.Focus group data was collected using Elluminate Live!, a web-based conferencing, video, voice, and file share system.Although the focus group sessions were scheduled for one-hour, each focus group lasted from 1.5 to 2 hours, even when a smaller number of participants were present.Focus group recordings were transcribed and a content analysis of the participant responses was performed.

Results
Evaluation Skills Self-Assessment Frequency analysis of ESSA pre-tests for those participating in the first three years of the project (N = 21) revealed that at the start of the project most were "Unfamiliar" (0 on the 0-3 scale) or simply "Understand the basic concept" (1 on the 0-3 scale) At initiation, participants were most familiar with skills related to "Communicating Results" (X = 1.63), with estimates of skills for Planning, Focusing, Design, Selecting Methods, Data Collection, and Analysis phases between 1.15 and 1.44/3.0.Over 50% of respondents rated themselves below the level of "Can Implement with Assistance" for most items.A large minority of respondents (40-49%) rated themselves below the "Can Implement" level on other skills such as Using a Logic Model, Determining Readiness for Evaluation, Developing Survey Questions, Matching Reporting to Audience Needs, and Developing Recommendations.A small minority of respondents (25-35%) rated themselves below the "Can Implement" level on only two items: Understanding the Purpose of Evaluation and Developing Evaluation Questions.
Subsequent analyses focused on continuing participants.Among the first and second year cohorts who completed an initial assessment and a one-year post test (N = 15), a clear trend toward higher scores on the ESSA was evident for all phases of evaluation.As indicated in Table 1, increases in six of seven categories were statistically significant despite the low numbers of participants.After one year of "learning by doing," over 50% of participants rated themselves in the "Can Implement" categories on 30 of 41 items (excepting Observation Methods and most Data Collection and Analysis items).
Self-ratings of the first cohort (N = 7; see Table 1) were lower at initiation and at the end of Year 1 than for the whole group (N = 15; see Table 2).However, as indicated in Table 2, cohort 1 self-ratings for most areas were significantly higher after Year 1 and after Year 2 relative to initiation.A majority of cohort 1 participants rated themselves as "Can Implement" on 28/41 items at the end of Year 1 and 32/41 items at the end of Year 2.
As noted in the project description, these professionals gained experience in designing and implementing a program evaluation or from critique and correction of evaluation design that was not ready to implement.Focus Groups Content analysis revealed three predominant categories related to the needs and challenges of evaluation capacity building: Organizational Culture, Evaluation Tools, and Evaluation Training Needs.The majority of focus group participants' experiences with evaluation were limited to onthe job and in-service training.
There were 76 comments reflective of Challenges to Evaluation; 48 comments related to Organizational Culture; 39 comments related to Evaluation Training Needs; 32 comments related to Evaluation Tools; and 41 comments related to the Benefits of E-LC Participation.The most frequently identified Challenge to Evaluation was time.In addition to time, communication and language difficulties, fear, and requirements for reporting were frequently mentioned.
Within the theme of Organizational Culture, participants most often discussed their need for cooperation, coordination, and support from state-and county-level leaders, the lack of time for evaluation work, and a perceived disconnect between reporting outcomes and the subsequent use of those reports.Learning Circle members reported that differing program emphases and evaluation standards among national, state, and county leaders regarding evaluation activities and outcomes made it difficult for them to identify and collect data on program outcomes.Many felt overwhelmed by expectations (perceived or real) to "evaluate everything," "evaluate something" (with insufficient guidance or flexibility on how evaluate), or "evaluate different things" (e.g., differing indicators at different levels).Their rapid work cycle and changing program priorities typically resulted in limited or no reflection on outcome reports.Within the theme of Evaluation Tools, focus group participants identified a need for packaged program templates to include built-in evaluation protocols and measures; an adaptable but consistent reporting system to capture short-and long-term outcomes; and state-created measurement tools for specific program areas.
The amount of variance in comments related to experience, goals, needs, and challenges was greater than the variance in the perceived benefits of participation in the E-LC groups.There were five clearly identified Benefits of E-LC participation that echoed from each focus group: 1) E-LC participants perceived an increase in the amount and quality of support from colleagues, 2) E-LC participants reported an increase in support and training from a state specialist, 3) E-LC participants identified an ability to better understand the need for and value of statewide program fidelity, 4) The ability to partner and collaborate with colleagues was enhanced through E-LC participation, and 5) Participants in the E-LC reported feeling more valued and validated with regards to their competency and qualifications to perform program evaluation in general.

Discussion
This evaluation capacity-building pilot project engaged 12 Extension 4-H youth professionals in Evaluation Learning Circles (E-LC) in the first year, another six in each succeeding year, with 21 of 24 participants still engaged in Year 3. Participants who joined E-LC teams tended to be internally motivated and self-directed in convening their groups, with the evaluation specialist acting as mentor and consultant rather than convener of E-LC teams.Most E-LC members began with relatively little training or experience in program evaluation.Team members recognized that building skills in programming and evaluation is a long-term process including concept and skill learning as well as hands-on trial and error.Teams met, on average, every 4-6 weeks.Active and maturing teams invested 10-20 hours in planning, implementation, and professional development between meetings while start-up groups generally gave five or less hours during the planning phase of their work.
All participants shared in developing logic models that some teams were able to implement as multi-county programs with evaluations.Some participants attended training webinars, conference workshops, and/or self-directed learning regarding evaluation skills and issues.Some consulted on specific programs or issues with an evaluation specialist.Thus the learning experiences of participants varied greatly in the first two years of the project.Nevertheless, having a "smorgasbord" of learning and project options seemed to facilitate learning across a multitude of schedules, learning styles, and program emphases.
Evaluation Skills Self-Assessment (ESSA) By their own assessment, E-LC members were relatively unfamiliar with evaluation competencies when they joined an E-LC team.The first cohort rated themselves lower than the second at initiation, which was somewhat surprising given their greater experience.Older, more experienced professionals may not have had the same opportunities to learn about evaluation as younger professionals but may have had more opportunities to recognize their need for training.Since the majority of Cohort 2 was pursuing advanced degrees, other learning experiences may have augmented learning in E-LC teams.However, graduate work also placed more demands on their time, so the practical, collaborative experiences may have provided a relevant, timely application of their classroom learning.
After one year, most participants indicated that they understood the basic concepts and skills involved in the preparatory phases of evaluation, but felt less capable in collecting and analyzing data and reporting results.This pattern is not surprising given that teams tended to spend the most time in learning and applying skills in preparation phases.Significantly, skill gains typically represented changes from "Unfamiliar" or "Understand Concepts" to "Can Implement with Assistance."Consistent with Arnold (2006) and Baughman, et al. (2010), shortterm mentoring in evaluation skills did not result in immediate competence and independence.Nevertheless, those in teams who completed projects reported the greatest gains in skills, with those who completed plans or logic models also citing positive gains.This correlation of accomplishments with perceptions is typical of project-based learning (Thomas, 2000).For practicing professionals, "learning by doing" proceeds somewhat slower than more structured approaches since it is typically "squeezed in" to professional schedules rather than "set apart" like a workshop or graduate course.More intensive training and mentoring might have yielded greater gains but may have resulted in greater attrition as demands exceeded available time.Moreover, the program relevance and rewards, including opportunities to report on program results, and personal mentoring and shared experience with colleagues represent key advantages of the "hands-on" approach.Practitioners' gains in the second year, continued participation into a third year, as well as addition of new teams each year illustrates the value they place on long-term mentoring and practice.
The present study represents a pilot project with a small, self-selected group of youth professionals engaged in learning at their own pace.Nevertheless, sustained and self-directed learning is unusual (Baughman, et al., 2010) and slow growth in evaluation competencies is typical (Arnold, 2006).Peer and expert support as well as practical, self-assessed gains in confidence and skills, and completed evaluation projects offer significant incentives for participation.Results underline the importance of sustained engagement for iterative skillbuilding and team-building.Team members who built trust and understanding of key concepts and valued projects over an extended time reported the most changes to practice and to continued participation in an E-LC team.The climate and outcomes fostered by E-LC teams provide a promising model for youth organizations to build evaluation skills on the job.Capacity-building efforts might also benefit from testing, expert observation, and more in-depth self-reflection.These methods might provide a more accurate and detailed record of changes, noting not only progress in skill-building but capacity to apply knowledge and skills as a guided or independent learner.Much valuable insight might be gleaned from more careful examination how E-LC teams learning and interact through hands-on projects.More detailed examination of organizational dynamics and change would provide insight on the context and effects of E-LC team development.

Limitations
This study is a pilot project with a convenience sample, involving no organizational incentives and operating with relatively few structured learning activities.Outcomes and experiences reported reflect a holistic learning process including training, mentoring, mutual assistance, and self-directed learning that are not assessed separately.At this point there is no standard to suggest what type and amount of learning activity is needed to achieve specific levels of competence.
Test effects consistent with repeated assessments, social desirability effects, and inexperience in judging evaluation capabilities could inflate or deflate estimates of personal competence.
Since the study included no independent assessment of skills, reported gains may reflect confidence resulting from project completion as well as genuine changes in knowledge or ability.However, project completion provides some evidence of competence, so scores are likely not simply a product of social desirability.Other outcome measures such as expert observation, interview, or judging of evaluation products might better corroborate or augment selfassessment and focus group results.

Implications
This report presents processes and outcomes of a pilot project with preliminary but promising implications for practice, research, and policy, as summarized below.
Practice A principal lesson of this study is that engagement of youth professionals in evaluation learning, like engagement of youth for evaluation, requires greater-than-average commitment but promises greater-than-average reward."Hand-selecting" team conveners and members who are experienced, motivated, respected, and capable of self-directed learning improves chances for sustainability.Enabling teams to select relevant projects enhances motivation and learning in-context, but teams may require guidance to meet evaluation standards of feasibility, utility, propriety, and accuracy (Yarborough, Shulha, Hopson, and Caruthers, 2011).
Learning circles may not work for everyone.Those best served will be able to commit to extended learning (vs. a short-term workshop), shared effort (vs.turnkey technology), and "going with the flow" and innovation (vs.packaged training).Team learning will require consensus, shared effort, and mutual support, which can be challenging for independent learners.Youth professionals who value immediate results may need short-term indicators of progress, for themselves and their project, in order to sustain engagement in team building and iterative learning.Those who value "learning by doing" will also need to invest some effort in concept learning to grasp the vocabulary, tools, and strategies that underlie program evaluation.A mentor-expert must be available, adaptable, accepting, astute to context and team dynamics as well as to evaluation issues.Team members should expect a mentor to be directive on skills and ethics but flexible in guiding other learning experiences.
Learning circle teams foster greater efficiency since they combine program planning and improvement, evaluation training, and accountability, including use of larger data pools, enable flexible scheduling and online resource availability.Best practices and ethics can be experienced, not just studied.A wide range of professional skills and youth or volunteer outcomes can be tracked in the context of ongoing programs across extended time.Such efforts are especially needed since there is little evidence on "what works best" for evaluation education.

Research
The ESSA identifies necessary evaluation competencies, but more research is needed to document typical skills of youth professionals or how, and how much, skills can be enhanced or applied over time.Focus groups, expert observations, and assessment of professional portfolios will add to understanding the goals, strategies and experiences, resources, and benefits of E-LC teams that contribute to sustainability, professional growth, and improved outcomes with volunteers and youth.

Policy
Agencies and professional organizations should consider the relatively low costs and high benefits, including relevance, convenience, empowerment, team building, and data pooling associated with supporting learning circles.Moreover, the model of field-generated organizational learning and capacity-building that might be applied across a variety of subjects and skills provides a valuable paradigm for continuing professional education in many youthserving organizations.
Participants identified several Evaluation Training Needs including a need for training on basic evaluation processes.Explicitly stated desire for information and training on when to use various types of evaluation, how to analyze and report outcomes, and how to use evaluation data to show program value were the predominant Evaluation Training Needs.

Table 1
Paired-Samples T-tests on Self-assessed Evaluation Skillsfor Long-term Learning Circle Members (N=7)

Table 2
Paired-Samples T-tests on Self-assessed Evaluation SkillsFor Mid-term Learning Circle Members (N=15)