Investigating the Utilization of Research Evidence in the 4-H Youth Development Program

This study investigated the acquisition, interpretation, and utilization of research evidence in the 4-H Youth Development Program from the frame of Social Cognitive Theory. Utilizing Consensual Qualitative Research, we interviewed twenty 4-H faculty, staff, and volunteers from seven states. Results indicated four domains, which covered participants’ definitions of research utilization, their experiences utilizing research, the process of acquiring and distributing research, and barriers and facilitators to research utilization. Participants described research use primarily in terms of improving 4-H programs. They discussed their level of confidence (i.e. self-efficacy) in finding and applying research evidence and their beliefs about the outcomes of research utilization (i.e. outcomes expectancy). Participants mentioned such barriers as not knowing where to look for research, lack of time, lack of funding, and difficulty applying research findings to their work. The facilitators included support from other 4-H colleagues and availability of 4-H specific conferences, publications, and curriculum databases.


Introduction
With a growing focus on the evidence-based practices, it is becoming increasingly important for organizations to successfully utilize research evidence and adopt research-based approaches.This is also true for the 4-H Youth Development Organization.As a result, we designed and executed a qualitative research study interviewing 4-H faculty, staff, and volunteers regarding their impressions of the utilization of research evidence in the 4-H program.In this introduction, we review recent dialogue on the gaps in the production and utilization of research, describe the relevance of 4-H to this discussion, and summarize findings related to utilization of research evidence.Cummings, Birdsell, & Norton, 2009) consider this to be a weakness and have called for studies that include additional variables such as organization and context (Davies & Nutley, 2008;Estabrooks et al., 2008).
Investigations that included a contextual component have suggested several barriers to utilization such as lack of time, autonomy, money, staff, access to research resources, and organizational commitment to research utilization (Barwick et al., 2008;Brown, Wickline, Ecoff, & Glaser, 2008).In contrast, the presence of learning opportunities, culture building, and availability and simplicity of resources were important facilitators (Brown, Wickline, Ecoff, & Glaser, 2008).In the Barwick et al. study, participants reported that desired resources included support for conferences, access to journals, and other professionals.
Given the contribution of individual and contextual variables to the prediction of research utilization, it made sense to us to employ Social Cognitive Theory (SCT; Bandura, 1986) as a guiding frame.SCT is an approach that is explicitly concerned with the interplay of social (e.g., organizational context) and cognitive factors in human functioning.There have been few attempts to examine research utilization and interest through the SCT lens.Bonetti and colleagues (2010) explored the use of evidence-based practice among general dental practitioners by applying a number of psychological theories.They found that SCT explained 29% of the variance in a behavioral simulation exercise and 16% of variance in statements of behavioral intentions.Relatedly, Bishop and Bieschke (1998) used an SCT frame to successfully model counseling psychology doctoral students' interest in research with variable sets representing research self-efficacy, research outcome expectations, environmental influences, and person inputs.Davies and Nutley (2008) have criticized previous studies of research utilization, indicating that they have often been uncritical and relied on surveys and self-reports.They called for, "more sophisticated examinations of knowledge engagement and knowledge interaction practices" (p. 6).Davies and Nutley elaborated on this point, suggesting that individuals may not be aware that they are utilizing research and calling for research that answers questions such as, "How can we track research use through unexpected avenues of diffusion?"and "What does research use look like at the individual, organization, and system levels?"(p. 7).Davies and Nutley (2008) distinguished between tracking forward and tracking backward approaches to investigating the utilization of research evidence.Approaches that use the tracking forward orientation examine how research findings make their way into practice and impact the individuals/communities they serve.In contrast, tracking backward approaches begin with practice behaviors to identify research-based influences.In this approach, the investigator attempts to disaggregate multivaried influences to discern the effect of research evidence.

Purpose
Consequently, as we designed our study, a) we committed ourselves to the SCT lens, b) we selected a qualitative strategy so that we could capture multiple perspectives, and c) with the goal of identifying multiple influences, we assumed a tracking backward strategy.
Qualitative research methods are useful in the early stages of a research project, as they can provide vivid descriptions of the phenomenon (Hill, Thompson, & Williams, 1997).Specifically, we used Consensual Qualitative Research (CQR; Hill et al., 1997) method because it offers a systematic and rigorous framework.In our proposal, the SCT constructs of research utilization self-efficacy, research outcome expectations, and contextual supports and barriers guided the creation of questions used to interview stakeholders (i.e.faculty, staff, and volunteers).Given the constructivist-interpretivist paradigm of qualitative research, we remained open to participant influences.That is, we encouraged participants to tell their story even if it fell outside of our SCT framework

Method Participants
The CQR approach, as outlined by Hill andcolleagues (1997, 2005), recommends a sample size between 8 and 15 and suggests that when data are collected from different sources there is the potential for a deeper and richer description of a phenomenon.Our sample included 20 participants: 4 were 4-H faculty (1 male, 3 females), 10 were staff (all female), and 6 were volunteers (all female).At the time of the interviews, participants ranged in age from 38 to 68 (M = 52.5,SD = 7.71) and previous involvement as a 4-H volunteer or a 4-H youth varied across participants (M = 10 years, SD = 11.6 and M = 6.5 years, SD = 7.2, respectively).
Participants came from seven states (i.e.AZ, GA, IL, MN, MO, OH, WA) and Washington D.C.In terms of ethnic or national background, 18 self-identified as Caucasian/White, 1 as Latino, and 1 as multi-ethnic.

Researchers
The first author was the principal investigator in this study; she designed the basic parameters of the study and initiated contact with our primary sources of recruitment.The first author has been with 4-H for 11 years in a role as a 4-Her and 10 years as a volunteer.The second author was the graduate assistant of the primary investigator and was involved in the study from its inception, assisting with data collection.To control for investigators' biases, the first two authors did not participate in the data analysis.The remaining investigators were doctoral students in industrial/organizational psychology programs enrolled in a class on qualitative research methods.

Interview questions
With the theoretical perspective of SCT in mind, the first two authors developed a list of start questions.The questions focused on several broad topics: participants' understanding of research within the organization, their familiarity in using and applying research evidence, how they access this evidence, and any barriers surrounding research access and use.Although scripted, the interviews were conducted in a softly-structured way such that the co-investigators were encouraged to follow the interviewee's lead in order to obtain a richer, thicker description of the participants' experience in using research.

Procedure Recruitment
Participants were recruited through purposive sampling and snowballing techniques.The principal investigator made initial contact within the 4-H organization to begin the snowballing process.We emailed each person the information about our study as well as a copy of our consent form.Upon receiving completed informed consent forms, we scheduled interviews.

Data collection
In order to maximize the number of participants involved in our study and allow for the participation of non-local parties, the interview protocol was conducted in the manner that was most convenient to the participant.The two interview methods were face-to-face and telephone interviews.For both interview methods, dialogue was transcribed as it occurred.
Two co-investigators were present during all the telephone interviews with one person asking the questions and fully engaging with the participant while the other person simultaneously transcribed the interview.

Data analysis
Researchers used the Consensual Qualitative Research (CQR; Hill et al., 1997Hill et al., , 2005) ) method to analyze the data.The CQR method specifies a series of procedures for coding data.First, the research team divided the responses from interviews into units that consisted of one complete thought.A unit could be as small as a single phrase or as long as a paragraph.
Second, looking at the units of data, team members suggested domains (or major themes) to which each unit would belong.After the teams determined that the domain list was sufficiently informative and inclusive, each team member independently read through individual interview transcripts and assigned each block of data to a domain.Any disagreements or inconsistencies that emerged during this process were resolved through consensus (hence the name of the research method) and, when necessary, the domain list was revised.
Third, after the teams had a stable list of domains, they summarized the content of each domain into core ideas (subthemes of a domain).In a manner similar to the development of the domains, the team members developed the core ideas independently, and then discussed these ideas until consensus was achieved.Following an audit (described below), the domains and core ideas were summarized into a CQR frequency table.

Trustworthiness
The trustworthiness of the project was supported in three ways.First, at the beginning of the study and each week thereafter, the researchers recorded and discussed biases and expectations that might influence the analysis (Hill et al., 1997).Our reflexive journal was maintained through an on-line discussion board, allowing for a dynamic exchange about the issues and ideas that emerged during the study.
Second, we presented our preliminary findings to a stakeholder group.Our stakeholders were selected for their knowledge about and investment in research utilization within 4-H.Following the presentation, we dialogued with our stakeholders about potential inaccuracies and future research possibilities.
Finally, we audited our domains and categories to determine that (a) the material was assigned to the correct domain, (b) the core ideas represented the content of the domain accurately, and (c) the wording of both the domains and the core ideas reflected the raw data well.The teams used the comments from both stages to revise domains and core ideas.

Results
The data collected from the participants represented various aspects of how research is used, understood, and valued in 4-H by volunteers, staff, and faculty nationwide.Examination of the data led to the identification of four domains: research value/definition, research utilization, research acquisition/distribution, and barriers/facilitators to research use.Readers are encouraged to read the narrative results in tandem with the CQR Frequency Table (Table 1).This simultaneously serves as an outline and an index of the salience of each theme.The table is organized by domains.Within each domain are the core ideas, organized in order of frequency-of-response.The Frequency column provides the narrative description: "general" means that the core idea is applicable to at least 75% of the participants; "typical" means that it is applicable to at least 50% (but less than 75%) of the participants; "variant" means that it was applicable at least 25% (but less than 50%) of the participants; and "rare" means that it was applicable to less than 25% of the cases.The Participants column provides the number of participants who mentioned the core idea.Finally, the Units column indicates the number of times the core idea has been mentioned throughout all of the interviews (without reference to the number of participants).These last two columns allow us to have some sense of how predominant a topic was for the participants.
For example, the category, 4-H is unique because of its commitment to research (located in the domain Research Value/Definition) is classified with the frequency of "typical."It was expressed by 12 participants (60%) and was identified in 32 interview units.That means that each of the participants mentioned this theme roughly three times.In contrast, the category, Preferences for non-research based approaches (located in the domain Research Value/Definition) is classified as "variant."It was expressed by five participants five times, meaning that each participant mentioned the theme once.
Quotations from the stakeholders are presented throughout the results.At times, these quotes have been amended to improve the readability of the manuscript.All changes have been made with care so that the quotes would still reflect the intent of the speaker.

Research value/definition
The definitions of research and research utilization varied widely.At one end of the continuum, a participant stated, "doing research gives you information."At the other end, a participant defined research as "based in logical positivism.There is one truth significantly regressed and disciplined within research.We're rigorous, reliable, etc.We deal with rejecting the null hypothesis."Many acknowledged "broadness" and "variability" in the definitions of research.
Generally, participants expressed the notion that 4-H is unique because of its commitment to research.That is, using and conducting research is a distinguishing characteristic of this youth program and increases its value.For example, one participant contemplated, "We need to be challenging ourselves.Participants attributed variation in the perceived importance of research to factors that clustered across three core ideas.Commonly, participants perceived that the importance of research increased at each level of the organization's structure.For example, a club-level volunteer stated, "I don't know that I see research as terribly important when I directly interact with the kids."In contrast, a faculty member stated, "the higher up you go in administration, the more important research utilization becomes.You become more responsible for ensuring that the staff seek out and apply research.At my level, I feel very responsible for getting research into programs."This perspective was repeated many times.
A second core idea related to the perceived importance of research suggested that it differs across roles in the 4-H organization.Central to this core idea were references to job descriptions and role definitions.Within this core idea, faculty participants frequently referenced "scholarship" requirements for promotion and tenure.Staff and volunteers were less likely to contribute to this category.
A third core idea related to the perceived importance of research indicated differences as a function of local organizational culture, individual differences, and time.As an example of organizational culture, several referenced differences in geographic regions.One stated it this way: "It is the environment we have and the expectations we have in our program.Things like that would make using research vary from state to state.The philosophy of their state's organization will impact how they look at the research."Regarding individual differences, several referenced the education level of volunteers, interest in the topic they are leading, and individual comfort with research.Some referenced changes as a function of time and funding sources.One stated it this way, "We went from a phase in the 70s where the research piece was emphasized, then it fell, now it's coming back.It seems to come with economy waves." In addition to organizational reasons for valuing research, several participants reported personal benefits from utilizing research evidence.For example, one participant mentioned, "Research helps me personally; I enjoy doing it and am personally comfortable doing it.I find it engaging and fun."A volunteer shared, "I've gotten a lot of free education as far as I'm concerned.Plus, it gives you a lot of self-worth too." Although it was a variant response, there was acknowledgement of those with preferences for non-research based approaches.One participant described this phenomenon: "I also run into people who say that they know what works with 4-H and our kids, and that we should not bother them with research.They tell us to stay away from their program, that they don't need us in here, they know what works."We add a note of interpretive caution to this category: no participant in this study endorsed this viewpoint.In all units in this core idea, the participants were referencing viewpoints of "others."

Research utilization
All Individual differences in comfort with research utilization were reported.Those who reported feeling efficacious in this area often qualified their statements.One added, "after years of experience," and another was comfortable, "in my own little small area." Almost half of the participants reported that evaluation data is important to stakeholders.The range of stakeholders was broad.A staff member stated, "If I have information on how to best engage a certain age group, I can pass that on to the volunteers, and they can program accordingly."Research also helps to obtain funding, "Some donors want to know if an experimental or a quasi-experimental evaluation was conducted by an external group.They will not give you any money until you can give that amount of rigor."Others referenced annual evaluations where the effectiveness of program data was assembled, reported, and evaluated.The results impacted program continuation, program improvement, and leveraging of funds from public and private investors.Some of the participants described a need to translate research findings into simpler, more practical language.For example, one participant's job involved "taking research results and conclusions and trying to turn them into practical ideas that people can use."Another participant reflected, "It's an important factor in research-presenting information in the practical, common use, so that the people using it can understand it.Otherwise, you're reading something that doesn't make sense." Almost half of the participants mentioned that situational factors influenced research use.
Examples included job requirements, personal interest, and so forth.One volunteer explained that her application of research was dependent upon the youth, "Not all research is applicable to all situations; every kid is an individual.If something isn't working, then let's do something else." A number of participants provided accounts of how they integrate experiences and nonresearch based knowledge to do their job.One participant put it this way, "It's a balance between what research says and pulling back to some of our traditions.For example, we still do the camp because we believe in our heart that it works.Now we have to do the research to prove what is in our hearts is true."Another stated, "I believe that dialogue, reading, and discussing how research is used with the youth helps, because then you can say that it worked in that county, and it may work in mine."

Research acquisition/distribution
Throughout the interviews, all of the participants discussed how individuals acquired and distributed research in the organization.The most common approach to acquiring and distributing information was the informal sharing of information with others -word of mouth.
One of the 4-H staff representatives said, "I think [volunteers] learn informally.I have not seen them in many classes, but I have seen them in my office a lot.They get it through networking and discussing.There is a lot of informal dialogue about what might work."Similarly, according to one of the volunteers, connecting with others is "very important.It's a network; everybody is helping each other." Searching the internet and reading journals to acquire information were also important.Both of these require some effort.One stated, "I do Google a lot, and it is figuring out the right word combinations to use."Another participant mentioned that there are journals available to staff and volunteers, "We have some journals that we can subscribe to and read, and also we can find research in other journals." Half of the participants reported using the 4-H curriculum and resources.Increasingly, this curriculum is expanding beyond the traditional project books and is accessible through the internet or electronic resources.One participant commented, "The research gets to volunteers in the form of curriculum developed from professionals based on the research."Similarly, a volunteer pointed out that "a lot of information comes through 4-H project books." Approximately, half of the participants regularly conducted literature reviews to find the information that they need.For example, one participant referred to literature when "having a difficult time with something."Another participant was doing "a lot of research, and looking at different articles, and trying to stay up on recent articles." Half of the participants talked about the importance of attending workshops and training.One participant acknowledged, "All of the leadership programs are research based.At any leadership workshop they will talk about the research behind the information."Additionally, almost half of the participants participated in conferences both internal and external to 4-H, which permitted them to share and acquire information.One of the participants expressed, "I've been to several conferences and leadership trainings, and they talk more about what research has shown us, this is how we work, it works."Another participant mentioned that research is shared "…at the national meeting.People do presentations, workshops, seminars, poster presentation, research presentations." Several acknowledged the importance of (and need to) utilize multiple sources.Although this core idea is identified as rare in Table 1, this designation is probably misleading.Most of the participants described multiple sources of information and these were singularly categorized in the preceding core ideas.One participant succinctly stated this core idea as, "We get our information from lots of places."Finally, although it was rare, a few described acquiring information by conducting internal research.They described creating/administering surveys and working "through the IRB process and learning to feel comfortable with that."

Barriers/facilitators
In spite of strong organizational commitments to utilizing research, the participants faced obstacles.One of the major barriers to research utilization reported by participants was a lack of time to acquire, utilize, and disseminate research.A curious pattern in this core idea was the recognition of lack of time experienced by the stakeholders at the organization level a step lower than the speaker.For example, a state-level faculty member stated, "We encourage county staff to read articles, to go to conferences, but they have a very, very full plate.We all have plenty to do."In parallel, a county level extension educator stated, "There is time challenge when you work with volunteer organization.Getting research out to volunteers in a useful, time respecting way is a challenge."Lack of research skills was mentioned almost as of often as the lack of time.A strongly and broadly expressed sentiment in this core idea was the need to have state and county faculty members who could translate research into accessible "terminology" with "practical application."One county level faculty member stated this with some passion, "A few years ago, some people sent out articles to people -not one iota of it was useful.I wanted the research summarized, and I wanted to understand its application."Participants talked about the difficulty of employing faculty/staff who conduct and understand research and interpret it in a way that the volunteers can easily understand and use.As one echoed, "Not everybody who is a good researcher is a good translator of it into practical application and use.You really have to search for people with some special skills to be able to do the translation.The skill set that it takes is really a challenge."Additional skills that were strongly desired included assistance "integrating" research into programs and activities; assistance with conducting needs assessments and program evaluations; and training for volunteers who were obtaining information on the internet.Specifically, there were concerns about skills and abilities in evaluating internet-based "research." Difficulties accessing research presented another common concern.Some sought the "source" of research; others had difficulty discerning what is "true" or credible research (particularly on the internet); and others desired, "a summary."A few individuals described logistical barriers to accessing research such as an inability to access the on-line database at the land grant university.In one instance this was coupled with a "9-hour drive to the library."In describing these obstacles, the participants desired "more accessible research structures," or a designated person.Many participants noted the interaction of this core idea with time.For some, this resulted in a lack of motivation to vigorously pursue potentially relevant research evidence.
In these economically difficult times, it was not surprising that several participants identified insufficient funding as another obstacle.One of the participants described, "We are in a tough budget time in terms of education and human service funding.We had significant cuts in staff and program dollars.When that happens and people are in the survival mode, they do not want to think about how to turn research into practice, they think about how to make sure the program survives."Regarding the difficulties in accessing research (the immediately preceding core idea), one participant referenced the "lost funding and the loss of our specialist positions." Almost half of the participants identified varied organizational barriers.Some discussed system and policy issues that impact the utilization of research.One described differences in educational requirements (e.g., bachelors versus masters) in county level faculty positions.Others described differences in professional development programs.Yet, others indicated recent or current organizational restructuring.Not surprisingly, many of these statements were connected to statements about funding.
Almost one third of the participants mentioned that individual resistance, particularly resistance to change, hinders research utilization.One of the participants said, "The biggest barrier is 'I've always done it this way, I'm not thinking about changing.'It's easier to do it the way you've always done it." A few participants mentioned insufficient research sharing.Included in this core idea was the desire for more programs to publish their research findings and the desire for a common source of relevant information.One stated the need this way, "We might be trying to do something new, but people may have already done it in the past." Finally, a few of the participants shared experiences with difficulties in applying research evidence.Specifically, participants suggested that the applications of research results were not always clear, or that the suggested application was not always feasible.One stated, "Academic journals generally don't tell you the application.They give you the information, but they don't go into how it applies.There needs to be a layer in there of application.How does it affect the 4-H program?"

Discussion
The results of this study provided useful information for understanding the barriers and facilitators of research utilization throughout the 4-H organization.Our project and findings present a unique contribution in that they broaden the view of the 4-H program to discover how research is incorporated into the organization.Because our study examined the experiences of faculty, staff, and volunteers, we were able to obtain a multi-angled perspective regarding the utilization of research, the outcomes of research utilization, and the contextual supports of research use at different levels of the organization.In the remainder of our Discussion, we connect our findings back to SCT, review the limitations of the study, and identify next steps.

Outcomes Related to Social Cognitive Theory
Consistent with previous research, (e.g. Brown et al., 2008;Meijers et al., 2006), we found that contextual factors such as access to research and resources and support for conducting and utilizing research affected research utilization.Specifically, participants talked about the barriers such as not knowing where to look for research, not having enough resources, lack of time, and difficulty obtaining the materials if they were not available online; these were similar to barriers previously reported in literature (e.g., Barwick et al., 2008;Brown, Wickline, Ecoff, & Glaser, 2008).Lack of funding was another concern that has been previously reported in research (Barwick et al., 2008).Finally, participants talked about the difficulty of applying research due to a lack of application suggestions in journals.Some of the supports included aspects such as the availability of conferences and 4-H specific publications.Additionally, 4-H colleagues seemed to play an important role by providing interpersonal assistance and needed information.Overall, the supports we identified were similar to those reported by Barwick and colleagues.In general, those participants who knew how and where to access information reported that 4-H provides an abundance of helpful materials.
Our results also supported the notion of a range of research utilization self-efficacy (e.g., beliefs about one's ability to perform specific behaviors or courses of action regarding research utilization (Bandura, 1986).Participants who voiced lower self-efficacy desired training/practice in locating research.In contrast, participants with higher levels of self-efficacy expressed confidence in integrating research evidence into their work.
Regarding research outcomes expectations, which involve beliefs about the outcomes of various courses of action (Lent, Brown, Schmidt, Brenner, & Treistman, 2003), we found that several participants had positive outcome expectations of using research.For instance, some said that research utilization would help build stronger communities.Others said it would provide better quality programs to kids and will improve the volunteer system of the 4-H program.

Limitations
Although the research design of our project is consistent with the standards for CQR (see Hill et al., 1997Hill et al., , 2002)), the CQR method has limitations.Because its origins are grounded in a constructivist-interpretivist philosophy of science, generalization of the results is not a goal.The constructivist position holds that there are, "multiple, equally valid, socially constructed versions of 'the truth'" (Hill et al., 2007, p. 197).Consequently, our results are affected by the personalities and interactions of the participants and researchers.Moreover, the voluntary nature of the project likely contributed to a selection bias.For example, our findings are based on the interviews with those who felt comfortable with the term, "research utilization"; the 4-H volunteers (as opposed to faculty and staff) in our project came from only two states; the majority of our participants were female; and all of our participants were recruited by e-mail (suggesting that all had some type of internet access).

Next Steps
We do not believe we are finished.So that we may evaluate research utilization within 4-H more formally, we are planning to use the results of this study to inform the creation of instruments to measure research utilization self-efficacy, research outcome expectations, contextual supports and barriers, and research evidence utilization.Overall, we hope that by utilizing multiple stages and multiple approaches our project and findings will help broaden our understanding of how research is incorporated into and utilized throughout the organization.
participants talked about what research utilization means to them, how they use research and research evidence in their work, and how research is used in the organization.More than half of the participants discussed how research is used to develop training, workshops, and program materials, providing numerous specific examples.Within this category there appeared to be several clusters of research utilization including project-specific program development (e.g., foods, animals, plant sciences), volunteer training, leadership development, and more general applications in working with youth.

Table 1
Summary of Domains, Core Ideas, and Frequencies What is that research base and what is it telling us?It is the fact that our program is built on positive youth development research that distinguishes us."Another participant highlighted the uniqueness of 4-H, "The research is what makes us unique to other groups."Many identified research as useful for improving the 4-H programs.Several described the importance of looking to research in developing programs and activities.Specific examples included using research to guide volunteer recruitment/selection, incorporating experiential learning, selecting age-appropriate activities, and developing curriculum.Others mentioned the importance of conducting program evaluation and using the results to determine program effectiveness and guide program improvement."On the program evaluation end we ask, 'How does the program work?' 'Is it successful?'We have to do that program evaluation."