An Experiential Model of Evaluation: Incorporating Graduate Students in the Collection and Reporting of Mixed-Method Data

A graduate level course was developed which incorporated the experiential learning philosophy of “learning by doing.” Students gained knowledge about program development and evaluation through using an on-line course management system as well as practicing what they learned during on-site evaluation visits. All background and supplementary reading materials, assignments, chats, and discussions were managed on-line. Students applied what they learned and collected quantitative and qualitative data while conducting site visits throughout the semester. Through this experiential learning evaluation course students made meaning from their direct experiences, took time to reflect upon their learning, and felt confident in their abilities as future evaluators. Having a number of graduate students from a variety of backgrounds added a breadth of content at little or no extra cost. This additional data can be used with program staff and stakeholders for continuous program improvement.


Introduction
"For the things we have to learn before we can do them, we learn by doing them."~Aristotle Because of their expertise, faculty are often asked do outside activities such as evaluating programs in addition to conducting their research, teaching, service, and outreach responsibilities.Having outreach opportunities to connect research and practice and then evaluate the outcomes is not only beneficial to the faculty person, program personnel and stakeholders; it can be beneficial in instructing graduate students as well.
Based upon the earlier contributions of John Dewey's philosophical pragmatism, Kurt Lewin's integration of theory and practice, and Jean Piaget's reasoning that intelligence is shaped by experience; a graduate level program evaluation course was developed.The course incorporated David A. Kolb's experiential learning theory whereby students were "learning by doing" (Kolb, 1984).Students gained knowledge about program development and evaluation through assigned readings, and then applied what they learned through conducting program site visits throughout the semester.Through an experiential learning method of evaluation, the course helped students make meaning from their direct experiences, as well as provide time and opportunity to reflect upon their learning.

Experiential Learning
Kolb's philosophy that knowledge is continuously gained through both personal and environmental experiences was the theoretical underpinnings for designing the course.According to Kolb, 1) the learner must be willing to be actively involved in the experience (concrete experience); 2) the learner must be able to reflect on the experience (reflective observation); 3) the learner must possess and use analytical skills to conceptualize the experience (abstract conceptualization); and, 4) the learner must possess decision making and problem solving skills in order to use the new ideas gained from the experience (active experimentation).(Merriam, Caffarella, & Baumgartner, 2007).
After completing the course, the program development and evaluation experiential activities enabled the students to: 1) understand federal requirements in a new start-up program in youth development; 2) understand and complete the university's institutional review board requirements for collecting data; 3) understand the need for and purpose of program evaluation; 4) understand the different methods of evaluation; 5) develop and complete an evaluation plan; 6) understand how to and complete on-site quantitative and qualitative data collection; and, 7) complete a professionally written evaluation report that can be submitted to all stakeholders as well as to the professor for student evaluation.

Course Logistics
In order for the graduate students to be comfortable with the program and complete an effective evaluation of the program, they had to understand the culture of each community, the purpose and nature of the youth programming, and the expected short and long-term outcomes of the program.All supplementary material was scanned and linked in an on-line learning system.For the initial site visit, the course instructor rented a university vehicle and transported the graduate students to the program sites.The time spent traveling together allowed for additional fact-to-fact instructional time as well as time for questions to be answered.Prior to the site visits, students were involved in learning how to apply for institutional review board approval for participant privacy.The desired short-term results or outcomes of the program indicated that youth who participated in the afterschool program will have: 1) increased communication skills; 2) increased self-responsibility skills; and, 3) increased healthy lifestyle choices as related to an increase in their physical activity and increase in selecting nutritious snacks.Long-term results of the program were indicated as an increased community capacity to sustain the afterschool program in the community.
Communities were selected based upon the following criteria: reaching underserved audiences with emphasis on Hispanic populations, risk factors including percentage of children at or below poverty level, schools with low academic performance, high unemployment rate, number of community partners, experience in offering afterschool programming, and Extension leadership.The model used for evaluation of the CYFAR SCP project was the Logic Model (United Way, 1996) which identified needs and assets, desired results both short-term and long-term, indicators which established how each outcome is measured, and the necessary activities and resources so that each outcome was met.

Data Collection
The evaluation encompassed the collection of both quantitative and qualitative data.Graduate students collected quantitative, or life-skills data, using a preapproved pre-and post-survey instrument.The graduate students administered the survey at the beginning of the school year which was the beginning of fall semester.Another set of graduate students administered the post-survey at the end of the school year which was the end of the spring semester.Clicker technology was incorporated to aid in efficiency of collection.Qualitative data was collected throughout the entire school year through observations and interviews of staff, children attending the program, and parents.
Using a number of graduate students to conduct the evaluation has shown a breadth of useful content.First of all, the age range of the children was kindergarten through eighth grade.
Therefore, the reading levels and abilities of each child varied considerably.Having more evaluators onsite allowed for additional one-to-one communication opportunities.This was especially beneficial for capturing the survey tool data.For example, if the student had difficulty reading or interpreting the question, a graduate student was present to read the question to the student.Collection varied across sites due to the number of children.Where participant levels were high, there may have been up to four students in one group.In programs where participants were of higher cognitive need, the survey was administered one-to-one.
Of special significance in using graduate students as evaluators was the collection of qualitative data.Having a variety of perspectives and a variance in content expertise such as child development, youth development, and nutrition resulted in more complex data that is less likely to be captured through one evaluator.The spontaneity and richness of the individual interviews with children, program staff, and parents added another dimension of capturing program particulars.The observations that were gained with many more sets of eyes were especially helpful in learning about peer-to-peer, staff-to-child, and other interactions.In addition, graduate students may take a stronger interest in the program and use the data for a master's thesis or project.In these particular experiences, the students traveled to the sites more frequently than two times per year.Therefore, the students became welcome guests at the program sites which in turn assisted the sites with more staff resources.
Lastly, the cost savings of using an experiential model incorporating graduate students in the program evaluation was noteworthy.No longer was the program paying one person for what can be a subjective review of one or two points in time.The graduate students had a vested interest in the evaluation experience because they were receiving graduate credit and actual grades from the supervising faculty person.Graduate students were required to write up final reports of each site in the format used by the project director and evaluator for end of year reporting to the funding agent.Since there was faculty supervision, graduate students took the extra time to be actively involved, reflected on their experiences, conceptualized their experience, and also used the new ideas gained from their experience.

Conclusion
Graduate students truly embodied the experiential learning model and "learned by doing" while still producing concrete useable final reports.They focused on the strengths of the program as well as looked for areas of improvement.To replicate this program design for evaluation, programs must be creative and flexible.The program sites were anywhere from 20 minutes to three and a half hours from the college campus.The supervising professor built in entire days in the beginning and end of the semester so graduate students knew the expectation for time and travel.All students were required to complete the university required insurance documentation prior to travel.While driving, the supervising professor was able to educate the students regarding each program site, and was available to answer the students' questions from the readings, program specifics, etc. Transportation was provided by a university-owned vehicle and driven by the supervising professor.The only additional expense for travel was the fuel cost which was paid from the CYFAR SCP project budget.
CYFAR SCP Project and the Logic Model In this case, the program being evaluated was a statewide Children, Youth and Families At-Risk (CYFAR) Program, Sustainable Community Projects (SCP), with five-year funding from NIFA/USDA.The 4-H Youth Development plan-of-work, supported by state administrators of Extension and the College of Agriculture and Life Sciences, identified six goals to enhance and develop life skills in children and youth across the state.Three of the six plan-of-work goals are addressed in the CYFAR SCP project which were: 1) Expanding Science and Technology; 2) Healthy Lifestyles; and, 3) Reaching Underserved Audiences.The CYFAR SCP Outcome was school-aged children, and the program model was 4-H Youth Development -Afterschool which addressed the physical, cognitive, social, and emotional development based upon research and science.4-H Afterschool is grounded in the 4-H Essential Elements of positive youth development, providing children with a sense of belonging and opportunities for mastery, independence and generosity.
They made recommendations based upon their content knowledge.The extra pieces of evaluation provided by the graduate students were insights that program staff and stakeholders can look into further and use for continuous program improvement.Having the experience of conducting an actual program evaluation not only satisfied a graduate credit requirement; it trained the graduate students on how to conduct a program evaluation for the CYFAR SCP 4-H Afterschool program.