Effect of HOBY Leadership Seminar on Self-Reported Psychosocial Outcomes in Adolescents

Civic engagement is important for the positive development of adolescents. As such, many youth development programs promote civic engagement, particularly community service and volunteerism. This report is a program evaluation of a youth leadership seminar that seeks to empower adolescents to engage in community service. Using a pre to post mixed-methods design, we evaluated 114 adolescent participants on several psychosocial outcomes. Findings indicated that participants experienced positive change that was consistent with program goals. Quantitative findings demonstrated self-reported increases in areas of cognitive autonomy, moral ideal and social responsibility. Qualitative findings included the important themes of an increased desire to volunteer, confidence, and positive feelings towards self. Implications of these findings and future directions are also discussed.

Because of these positive outcomes, volunteerism is considered by many to be important for positive adolescent development, so many programs incorporate volunteerism into their curriculum (Alt, 1997).Evaluations of such programs report increases in awareness of social issues and social responsibility (Lee, Olszewski-Kubilius, Donahue, & Weimholt, 2008), personal competence (Newmann & Rutter, 1983), decision-making power (Hamilton & Fenzel, 1988), prosocial behavior, and future civic engagement (Reinders & Youniss, 2006).It is important to note, however, that programs that use volunteerism as part of their curriculum have differential outcomes.One study suggests that when a program engages students' sense of citizenship, as a personal virtue, and promotes personal efficacy it is more likely to promote social responsibility and civic engagement (Ballard, 2014).In an effort to further understand the impact of programs that use volunteerism as a component of their positive youth development curriculum, this report is an evaluation of one such youth development program.

HOBY State Seminar Curriculum
Hugh O'Brian Youth Leadership (HOBY) is a non-profit organization that provides leadership education to high school students across the United States as well as internationally.HOBY's main offering is a 3-to 4-day state leadership seminar that is attended by exemplary students selected by their high school administration and teachers.There are currently 70 different sites across the United States that host these state leadership seminars.HOBY's various leadership programs, including the state leadership seminar, use the Social Change Model of leadership (SCM) as a foundation for its curriculum (Ferrence Ray, 2016).Two key concepts of the SCM are that the ultimate goal of leadership should be to accomplish positive social change and that a leader need not possess a formal position to exercise leadership (Wagner, 2007).The SCM divides leadership development into three domains, each consisting of certain values.The first domain is composed of individual values of consciousness of self, congruence, and commitment.Within this domain, leaders need to develop an awareness of their values, strengths and weaknesses, feelings, and beliefs.They need to live in a way that is congruent with that self-awareness, and they need to be committed to act on those values and beliefs.The second domain contains the group values of collaboration, common purpose, and controversy with civility.These values represent interpersonal skills that are necessary for effective leadership towards the goal of social change.The final domain contains the societal value of citizenship.Citizenship represents leaders' connection to their community.This connection fosters a sense of responsibility to act for the good of that community.As leaders fosters the values within these three domains, they are in a position to engage in positive social change, which is an important goal of leadership (Komives & Wagner, 2009).
Using the SCM as a foundation, the HOBY state leadership seminar's curriculum is broken into four modules.As seen in Table 1, the first three modules correspond with the three domains of the SCM and the fourth module focuses on social change through volunteerism.Each module consists of a module introduction, a speaker or panel discussion, and an activity which is designed to promote development of the values set forth in the SCM.During the fourth module, each seminar site prepares a service project that the participants complete.The culminating event of the final module is a service project.Each seminar site implements the curriculum differently using different activities and speakers based on local needs, but the curriculum structure is the same across sites (for additional information on HOBY and its curriculum, see Ferrence Ray, 2016).In an effort to evaluate the HOBY state seminar, we sought to determine whether seminar participants experienced changes in several psychosocial

Cognitive Autonomy
Cognitive autonomy was selected because its various subscales capture aspects of the virtues associated with the first two modules of the HOBY curriculum.Cognitive autonomy was measured using the Cognitive Autonomy and Self-Evaluation (CASE) inventory (Beckert, 2007), which contains 27 items measured using a 5-point Likert scale ranging from 1 (strongly agree) to 5 (strongly disagree).

Moral Ideal
Moral ideal self, selected because it is a construct that describes the kind of moral person an individual would like to be, was measured using the Moral Ideal Self Scale (Hardy, Walker, Olsen, Woodbury, & Hickman, 2014).Participants ranked 30 attributes, 20 moral-ideal-self items (e.g., does good actions, generous, good example, compassionate) and 10 distractor items (e.g., energetic, organized, good sense of humor) using a 7-point Likert scale ranging from 1 (not at all) to 7 (very much).Before completing the scale, participants were given the prompt: "When you think about the future, what do you want yourself to be like?"In addition to ranking each of the 30 attributes, participants were asked to list the eight attributes that they felt were the most important to them at their current stage in life.While Hardy and colleagues suggest that the items consist of a single scale, a principle component analysis of the postseminar data revealed three components that we labeled integrity, altruism, and character.
These three components were used as subscales of the Moral Ideal Self Scale in subsequent analyses (see Table 2).The data in each of these components demonstrated good reliability: Integrity (α 1 = .90,α 2 = .90),Altruism (α 1 = .89,α 2 = .89),Character (α 1 = .74,α 2 = .74).Three items (generous, does good actions, and forgiving) loaded similarly on two factors and thus were excluded in subsequent analyses.The integrity domain, within the Moral Ideal Self Scale, aligns with module 1 of the HOBY curriculum while the altruism domain aligns within module 3. The character domain does not align with a specific aspect of the HOBY curriculum.

Social Responsibility
Social responsibility was selected to measure growth in relation to module 3 of the HOBY curriculum, which is designed to promote citizenship and participants' connectedness to their community.To measure social responsibility, we used the Youth Social Responsibility Scale -Short Form (Pancer, Pratt, Hunsberger, & Alisat, 2007) which contains 10 items (e.g., "Teenagers should just enjoy themselves and not worry about things like poverty and the environment") measured using a 5-point Likert scale ranging from 1 (strongly agree) to 5 (strongly disagree).Coefficient alphas for these data demonstrated adequate internal consistency (α 1 = .72,α 2 = .63).

Short-Answer Questions
In addition to the survey items, in the post-seminar questionnaire we included three shortanswer questions for participant consideration: (a) How has this seminar influenced your desire to volunteer in the future?, (b) How has the seminar influenced the way you see yourself (your sense of self or identity)?, and (c) How will you live your life differently (think, act, feel) now that HOBY is over?The first question was included to assess development in module 4 of the HOBY curriculum which focuses on encouraging civic engagement through volunteerism.The last two questions were included to gain additional insight into aspects of participant change that was not included in the quantitative measures.

Mixed-Methods Analytic Strategy
Consistent with the mixed-methods concurrent triangulation design, we used paired samples t-tests and repeated measures ANOVAs to analyze the quantitative data and a summative content analysis to analyze the qualitative data.
The goal of summative content analysis is to describe word usage in order to understand the word's contextual meaning (Hsieh & Shannon, 2005).The first question asked participants how the seminar influenced their desire to volunteer in the future, and all participants endorsed an increased desire to volunteer.In response to the second question, participants related positive growth regarding how the seminar influenced their sense of self or identity.Finally, participants expressed motivations to self-improvement when asked how they would live their lives differently after HOBY was over.Because all of the participants reported positive change as a result of attending the seminar, the summative content analysis allowed us to analyze the different ways in which the participants felt that they had grown.Responses were initially coded using values coding.Values coding focuses on the values, attitudes, and beliefs of the participants by exploring the way that participants think and feel about and the importance that they place on themselves, others, and ideas (Saldaña, 2016).After the first cycle using values coding, pattern coding was employed to condense the responses into similar patterns of responses (Miles, Huberman, & Saldaña, 2014).

Quantitative Data
A two-way repeated measures ANOVA was used to test for any change in cognitive autonomy, from pretest to posttest, using time and the five subscales of the CASE inventory (see Table 3).
Since the data failed Mauchly's test of sphericity for the interaction between time and domain, w = .653,approximate χ 2 (9) = 37.26, p < .001, the Greenhouse-Geisser correction for degrees of freedom was used.This test showed no statistically significant interaction effect between time and domain.Investigating the main effects of time for each of the CASE subscales revealed significant changes in all but one.To reduce the likelihood of a Type I error, a Bonferonni correction for multiple comparisons was used resulting in a new critical value of p < .01.Participants showed the greatest change in levels of comparative validation from pretest to posttest.Other statistically significant changes included the subscales of evaluative thinking, decision making and voicing opinions..000 .000 .000 .000 .

323
.360 .032 A second two-way repeated measures ANOVA, using time and the three domains (integrity, altruism, and character) extracted from principle component analysis, examined change of moral ideal from pretest to posttest (see Table 4).Since these data also failed Mauchly's test of sphericity for the interaction between time and domain, w = .835,approximate χ 2 (2) = 15.18,p = .001,the Greenhouse-Geisser correction for degrees of freedom was again used.This test revealed a significant interaction between time and domain.Inspection of the estimated marginal means and post-hoc pairwise comparisons show that all three domains increased after the seminar.Despite having the highest mean during pretest and posttest, changes in the domain of character failed to reach statistical significance.The domain of altruism had the lowest pretest mean score and showed the greatest increase at posttest.Changes in the domain of integrity were also statistically significant from pretest to posttest.

Qualitative Data
Of the 114 participants, 94 completed the short answer questions.Each response aligned with at least one category that emerged from the pattern coding process.Some responses endorsed ideas from multiple categories, and in these cases, responses were counted in both categories.
Within each question, three categories emerged from the coding process.Final themes for each of the questions can be found in Table 5.
When participants were asked how the seminar had influenced their desire to volunteer in the future, their responses were classified as internal, altruistic, or generic.For those with internal motivations to volunteer, some related how volunteering made them feel good, "I didn't understand the amazing feeling you get [when you serve]," and others saw volunteering as a way towards self-improvement, "It helps us to become a better person.The people that talked to us helped us realize that I can become someone."Among those who expressed more altruistic reasons for volunteering, some participants stated a desire to help individuals, "I feel inspired to help those that really need it," and others recognized their part in society, "It makes The second open-ended question asked participants how the seminar had influenced their perceptions of themselves.Responses fell into categories of perceiving themselves more positively (or being a person of value), more capable, and having a greater self-awareness.
Those who saw themselves in a more positive light made statements such as, "[HOBY] has given me an understanding that I am worth something."These statements sometimes focused on not letting previous mistakes interfere with their current efforts, "My mistakes don't need to bog me down," and increased self-confidence, "Honestly, lately I'd been having some selfconfidence issues.This seminar has relieved my stress and made me find my confident self." Those who felt like they were more capable often spoke of their potential to make a difference in the world, "I feel like I can do anything.I can be a leader and make a difference."Lastly, those who found greater self-awareness mentioned feeling more clarity about oneself, "I think I see myself more clearly."and recognition of qualities that were previously unrecognized, "I have discovered new things I'm good at." Finally, when asked how they would live their life differently (think, act, feel) after HOBY, participants' responses were grouped into three categories: a desire for self-improvement, a desire to be more kind and considerate to others, and an increased desire to volunteer.
Statements showing a desire to improve could be either general, "I will focus less on what is popular.I want to be outstanding rather than normal," or talked about specific skills they had learned from the seminar that they wanted to implement when they got home, as in this student's reference to one speaker's advice on how to set and accomplish goals: "I will set goals and follow the six steps to accomplish them."Statements such as "I will be kinder and more understanding of others" exemplified the second category.Those who wanted to volunteer more expressed plans to join more service organizations: "I am definitely going to sign up for more service clubs" or a general desire to give more service: "I will start serving whenever I can."

Discussion
The purpose of this study was to determine whether participants of a HOBY state seminar experienced changes in various psychosocial domains consistent with program goals.HOBY bases its curriculum on the Social Change Model of leadership and so one would expect that participants would develop greater understanding and commitment to personal values, greater interpersonal skills in working with a team, a greater sense of citizenship and social responsibility, as well as a desire to be active in their community through volunteerism.The data presented here suggest that participants of this HOBY state seminar experienced psychosocial changes in line with program goals.
Module 1 of HOBY's curriculum focuses on encouraging participants to gain a greater understanding and commitment to personal values as they develop the SCM values of consciousness of self, congruence, and commitment.From the quantitative data, participants demonstrated improvements in the cognitive autonomy domain of comparative validation.This means that participants were less likely to feel as though they needed external approval for their decisions.Increases were also observed in evaluative thinking suggesting that participants are more likely to evaluate their own thoughts and actions.Self-evaluation, the final domain from the CASE inventory that aligned with this area of programing, increased from pretest to posttest, but the increase was not statistically significant.However, from the qualitative data we see that the majority of participants endorsed a greater awareness of self.This could mean that this domain within the CASE is measuring something different from what was uncovered through the short-answer questions.However, another explanation is that participants came to the seminar with one sense of self but after participating in the HOBY state seminar, their sense of self changed.One participant said, "I knew myself before I came but with this 'new [me]' I felt I could be as crazy as I wanted and that no one would judge."Like this student, it might be that participants felt that they knew themselves well before attending the seminar but the changes they experienced amounted to what they felt was a new but equally genuine self.This would account for the lack of significant change in the quantitative data and the endorsement of greater understanding of the self in the qualitative data.Finally, participants also showed improvements in those values within the integrity domain of the Moral Ideal Self Scale.This domain contains attributes such as "has good values," "follows values," and "stands up for his/her beliefs." Module 2 of the HOBY curriculum focuses on interpersonal skills in working with a group.This aligns with the SCM values of collaboration, common purpose, and controversy with civility.
None of our measures allowed for a robust evaluation of interpersonal skills.However, the voicing opinions and decision-making subscales of the CASE inventory represent qualities that best align with this module of the HOBY curriculum.Confidence in making decisions and a willingness to voice opinions are both valuable skills when working with a group.Participants in this program showed significant increases in these areas from pretest to posttest.
Module 3 of the HOBY curriculum focuses on developing the SCM value of citizenship.
Participants demonstrated increased social responsibility, as measured by the Youth Social Responsibility Scale, and outward focused values, demonstrated in the altruism domain of the Moral Ideal Self Scale.These results align with module 4 of the HOBY curriculum and highlight volunteerism.When asked how the seminar has influenced their desire to volunteer, 100% of respondents endorsed an increase in their desire to serve.These were sometimes generic declarations of a desire to serve, but many participants expressed both internal and altruistic reasons for wanting to increase their volunteerism.The theme of volunteering was also found in the responses to the question about how participants were going to live their lives differently after the seminar.
Finally, there were several findings that did not align with the goals of the HOBY curriculum but that still highlighted positive psychosocial outcomes.Participants showed that they developed feelings of self-worth and confidence in their abilities.They also demonstrated a desire to be kinder to others and were motivated towards self-improvement.

Limitations
While the findings from this program analysis are promising, it is important to recognize the limitations of this research.First, this research focused on a standalone experience not connected to further programming in HOBY.These findings are limited by the lack of longitudinal data to see if these changes are stable across time.Participants of HOBY often describe what they call a "HOBY high" after the seminar ends; like the participants in this study, they express confidence in their abilities to make a difference in the world and accomplish great things.These feelings likely come from the high energy of the seminar and interacting with other youth who are also motivated to do great things.It is unclear whether these positive changes remain stable after participants return to their daily routines.We would recommend a bit longer delay in post-program assessment to evaluate the staying power of the initial positive effects of the program.While not robust, a small number of participants (n = 15) from this sample agreed to complete a 12-month follow-up questionnaire with the same quantitative measures.These participants showed a regression of their scores from posttest levels but all scores remained above pretest levels.These findings are promising but we use caution in interpreting this finding, which could be due to a selection effect of those who were willing to complete the follow-up questionnaire.
Another limitation from this study was the absence of any comparison or control group.This limitation opens these findings to effects of history or maturation and limits generalizability.
However, maturation is unlikely to have occurred since the length of the seminar was only three days and history effects are equally unlikely since participants where engaged in programming for its entirety except for during meals and sleeping time.As the purpose of this study was to evaluate HOBY's program, generalizability was not an expected outcome.However, especially for similar programs that utilize the Social Change Model of leadership, these preliminary findings can provide an important first step in understanding the effects of the initial interventions.
me want to give back to my community."One participant that expressed this altruistic motivation to volunteer said, "I am much more motivated to go out and do good in the world!I realize now how many opportunities there are to serve our fellow man.I am inspired to find something good to do with my life."Finally, those whose responses were categorized as generic expressed a desire to volunteer without specifying a motivation as to why.Generic response examples were statements such as, "[HOBY] has given me so many more ideas for volunteering" and "I'm already thinking about the service programs I'm going to join."Table5.Frequency of Themes in the Qualitative Analysis of the Shortknow that I can make a difference" "I feel like I know myself a lot clearer now" How will you live your life differently (think, act, feel) now that HOBY is over?strive to take advantage of every day" "I will be more considerate of others" "I am going to look for every opportunity to serve"

Table 4 . Two-Way Repeated Measures ANOVA for Moral Ideal
was used to test changes from pretest to posttest.Results showed a statistically significant difference between pretest and posttest.