A New System of Classifying Out-of-School Time Job Responsibilities

Out-of-school time (OST) is a burgeoning field with both research and policy implications. Efforts to improve professional development for OST staff members are of particular interest, as funding streams increasingly target interventions which promote positive changes in student outcomes. Professional development evaluation in particular is hindered by a lack of consistency among job titles and responsibilities across OST organizations. This mixed-method study utilizes original data to explore underlying patterns of job responsibilities within the field and offers a new classification system based on exploratory factor analyses. The classification includes five categories, each with a unique combination of common job responsibilities to assist survey respondents in choosing the appropriate category: upper-level administration, mid-level administration, directservice, capacity-building, and “other.” Results suggest this new system is user-friendly to both respondents and researchers, and will garner more accurate and comparative information for future OST research and application.

A growing area of emphasis within OST programming is professional development.This has been identified as a critical element of OST programs for a number of reasons.OST staff represent a variety of backgrounds and preparation, and include school teachers, teachers' aides, social workers, parents, and community members.Thus, staff need specific training to succeed in an OST settings.In addition, staff retention in OST programs is often challenging (Partnership for Afterschool Education, 1999;Bowie & Bronte-Tinkew, 2006) and professional development is one strategy for enriching and retaining staff.
Moreover, studies have linked professional development to positive student outcomes in both formal education and out-of-school time settings.In general, professional development for classroom teachers is associated with a variety of positive outcomes relating to student achievement, classroom management, classroom environment, etc. (The Public Education Network and The Finance Project, 2005).National Board Certification processes are also associated with the development of stronger curricula and teachers' increased ability to evaluate student learning (National Board for Professional Teaching Standards, 2001).Research suggests that targeted training is essential to establish this association.One experimental study found that classroom teachers attending training targeting specific outcomes and aligned with a specific reform initiative improved their classroom practice, while those attending trainings on more general topics showed no change in their practice (Whitehurst, 2002).
Research on professional development within OST programs also suggests positive outcomes, but focuses on different mechanisms and outcomes.For example, an evaluation of the Building Exemplary Systems for Training (BEST) initiative (Fancsali, 2002) found that professional development is an essential part of quality improvement and programming.Further, these results suggest that the most effect types of professional development are continuous trainings that span a diverse range of topic areas (such as youth development and how to provide quality programming).Similar to formal education, professional development in OST programs can positively impact student outcomes as well as staff retention (see Bouffard & Little, 2004 for an extensive review of the literature in this area).Bowie and Bronte-Tinkew (2006) extend this association to identify benefits to the individual youth worker, the program, and the OST field in general.
Professional development is an indispensable component of out-of-school time programming, and so is its evaluation.However, a lack of universal evaluation instruments utilized across the field has hampered both program evaluation (Geiger & Britsch, 2004) and professional development evaluation (Kane et al., 2006), but is particularly problematic for the latter.Without reliable instruments to assess professional development impact, it is difficult to truly ascertain the base knowledge level of the participants, increases in learning associated with professional development experiences, or optimal strategies for allocating future funding.Studies are underway to increase the validity and reliability of these instruments in a variety of settings (Kane, Peter & Gabel, 2008), but even these studies run into a common problem found in OST research: the field lacks a system of common job titles, descriptions, and associated responsibilities.
OST researchers are well aware of this problem and often find discrepancies between selfreported job titles and actual job responsibilities (Fusco, 2003;LeMenestrel & Dennehy, 2003).That is, specific job responsibilities under a given job title are not consistent across OST organizations and/or programs.For instance, in the OST field, Site Director, Program Director, and Afterschool Coordinator commonly describe the same position, including the same responsibilities.Due to the diversity of programs and organizations that fall under the umbrella of "out-of-school time," this variation in the language used to describe job titles and responsibilities is understandable.Yet this presents a problem to researchers who request job titles as part of demographic data.If there is little consistency among job titles within the field, then there may be minimal value in gathering this information.
Collecting accurate information on job descriptions is especially critical to the OST field, since previous research indicates that job responsibilities affect how participants respond to professional development experiences.Through a series of five focus groups (n=50 participants) in the fall of 2004, Kane et al. (2006) found that OST administrators often seek formal, informational workshops, whereas direct-service staff generally prefer interactive workshops.Thus, the manner in which staff are categorized, in terms of their job responsibilities, is related to different substantive topics and learning styles within professional development settings.Since staff respond differently to various workshop formats, it is clear that workshops should be designed and implemented to meet the unique needs of different audiences.However, the extent to which these efforts can be successfully implemented depends on the ability to accurately classify staff.

Pilot Survey: Data Collection and Analysis
In December 2005, we conducted a pilot survey to test a new system of collecting information on the job titles and responsibilities of OST staff members.This survey gathered two essential pieces of information.First, rather than requesting job titles to locate survey respondents within the field of OST programming, this new system categorized individuals according to their primary job responsibility as an indicator of their role in the field.We hypothesized that this study could serve as a valuable resource for researchers and evaluators who gather comparable information from national respondents who serve in a wide variety of settings (such as schoolbased, community-based, or faith-based programs) who may utilize different language to identify job titles and responsibilities.Second, participants were asked to identify all of their job responsibilities from a comprehensive list.This allowed for the exploration of patterns within the data that may not have been readily apparent.
This pilot consisted of both quantitative and qualitative data comprised of 231 online survey respondents and 110 interview respondents.The online survey was created by the authors and advertised through various local, statewide and national listservs.The qualitative interviews were conducted within a series of national focus groups coordinated by the National AfterSchool Association.In both settings, participants were first asked to identify a primary job responsibility from a list of fourteen options (see Figure 1), and then were redirected through a skip pattern to a job description corresponding with the identified primary job responsibility (see Figure 2).Participants were then asked if the job description "accurately summarized most of the job responsibilities" within their position.

Figure 1
Primary Job Responsibilities Used in the First Pilot Survey Please select your PRIMARY job responsibility from the following list (circle ONE): 1) Oversee all aspects of one or more organizations 2) Oversee multiple programs/sites within an organization 3) Results from both the survey and focus groups yielded two important findings.First, participants were able to select a primary job responsibility, indicating that the choices were clear and appropriate.Second, the vast majority of participants reported that the pre-specified job description was accurate in summarizing most of the responsibilities within their position (90.3%).Taken together, these indicate the potential success of this new system.Furthermore, since a strong link was successfully established between the primary job responsibilities and job descriptions, this demonstrates that researchers may be able to rely on the primary job responsibilities as a proxy for the relatively larger set of information encompassed within the job descriptions.Thus, we concluded that a simplified set of primary job responsibilities, for the vast majority of respondents, would successfully provide ample information about their overall job responsibilities in a more space-efficient manner for a survey format, while providing researchers with rich contextual information from the descriptions.
Subsequent analysis of the pilot data revealed another valuable finding-an underlying pattern existed within the groupings of job responsibilities.Survey respondents, regardless of their primary job responsibilities and job descriptions, performed many different tasks that overlapped between categories.For example, both teachers and administrators reported working directly with children and creating/developing programs or activities.As a result, we explored the possibility of using a unique combination of primary job responsibilities in order to more fully assess the roles of OST workers.This classification could serve to unify an even broader set of information gathered from OST workers through surveys and evaluations, while maintaining a strong basis for inter-organizational comparisons.
To do this, we performed an exploratory factor analysis (EFA), which can be a useful data reduction technique.It is advantageous in that it explores underlying clustering patterns in the data.This is especially useful for this area of research, given the lack of universal job categorization in the field.The results from the factor analysis showed that three factors, or clusters of job responsibilities, emerged.These included: Category The key word within each description is "or."These results indicated that three separate clusters of primary job responsibilities typically emerge, but they do not mandate that all tasks be performed in any single position.For example, individuals who fall under the "Administrators" category are those whose primary job responsibility is to manage a site or organization, manage budgets, write grants, fundraise, or work with a governing board.Not all administrators need to perform every one of these tasks in order to be categorized in this way.
Rather, this analysis shows that these types of responsibilities tend to be highly correlated with one another.Importantly, this provides some evidence that a more parsimonious classification system can be utilized to simplify the original list of primary job responsibilities and job descriptions.Several advantages exist with such categorization: it takes up less space on a written or online survey, it decreases the respondent's burden, it is easier for researchers to interpret three categories of individuals rather than fourteen, and it is easier for practitioners to analyze differences based on a smaller number of groupings.

Second Pilot Survey: Data Collection
To test these hypothesized groupings on a larger scale, we revised the pilot survey to reflect these three categories and released its second Job Title Survey in September 2007.The survey remained open for 6 weeks, and was advertised through several organizations (such as the National AfterSchool Association and the Pennsylvania Statewide Afterschool/Youth Development Network) and through multiple listservs (such as those distributed by Promising Practices in After School, SAC-L, and the authors' listserv).Overall, 1,390 individuals completed the survey.
In many respects, this convenience sample mirrored the field of afterschool workers (see Table 1).Similar to national estimates from a probability sample of the human services workforce examined by Light (2003), respondents to survey were mostly female (79%) and predominantly White (66%).A smaller percent were African American (17%) or Latino (7%), and the mean age of the sample was 42 years old.However, this sample diverged from Light's estimates in that the respondents in this survey represented a more highly educated sample.
Here, most held either a Bachelor's degree (39%) or a Master's degree (29 %), whereas in Light's sample most individuals had either some college (22%), or a college degree or higher (52%).In addition, in this survey half held some form of license or certification (such as for teaching or social work, 50%).In terms of job characteristics, most worked full-time (79%) and were employed by either a large organization (of 100+ employees, 42%) or a small organization (less than 50 employees, 30%).Midsize organizations were not as prevalent within the survey responses.Part-time workers worked an average of 26 hours per week, earned an average annual salary between $20,000 and $24,999, and did not generally receive any health benefits from their employers (69%).Full-time workers worked an average of 45 hours per week, earned an average annual salary between $45,000 and $49,999, and generally received either full health benefits (38%) or partially paid health benefits (47%) through their employers.A wide range of states participated -only 5 states in the US were not represented -although clusters of surveys came from New York (22%) and Pennsylvania (13%).Some international OST workers participated from countries such as Columbia, Germany, Japan, South Korea, and Singapore (although to maintain consistency regarding work environments we include only OST workers from the United States in our analyses).
Lastly and most importantly, this larger sample yielded a more equivalent representation across the given categories than the pilot sample.This provides further confidence in the estimation of each category.Specifically, 46% identified with Category 1 (Administrators), 24% with Category 2 (Program Staff), and 17% with Category 3 (Intermediary Staff).

Findings
Participants were given the option to choose one of the three categories that included their primary job responsibility, or a fourth option, "None of these adequately describe my job responsibility," which also allowed the participants to provide written explanations.Eighty-five percent of the respondents chose one of the three categories.While this indicates that most participants can successfully utilize this new system, three revisions can further increase its effectiveness.
First, a review of the reasons provided for those who chose "none of these adequately describe my job responsibility" showed that many respondents wanted to choose more than one category.Program directors and upper-level administrators, in particular, tended to respond that they were responsible for "all of the above."While it is unlikely that a given position includes every single primary job responsibility, it is clear that some staff had difficulty deciphering primary from secondary responsibilities.Therefore, a separate category could be included (e.g., "Other") that allows participants to write in different responses.When the writein responses were recoded to accommodate this change, 96% of respondents "fit" into one of the options provided.This suggests that this slight revision would significantly increase the efficiency of this system.
Second, the remaining 4% represented individuals with primary job responsibilities that were not listed among the three categories.Job titles within this omitted category included Administrative Assistant, Office Manager, Human Resources, Licensing Coordinator, Advocacy Worker, Technical Support Staff, and Resource/Referral Staff.It seems that while the existing three categories included enough detail to capture most of the variation of primary job responsibilities, the inclusion of a few extra responsibilities would further increase the efficiency of the system.
Third and most importantly, we performed another exploratory factor analysis on the comprehensive list of job responsibilities from this larger sample to see if it generated results similar to the first pilot (see a correlation matrix of responsibilities in Table 2 and results of the factor analysis in Table 3).

Limitations
As in any study, this is not without its limitations.This study relied on a convenience sample in which there was limited control over who completed the survey.The study was advertised through several organizations (e.g., The National AfterSchool Association and the Pennsylvania Statewide Afterschool/Youth Development Network) and through multiple listservs.Solely relying on organizations and listservs excludes OST professionals who may not belong to the National and State organizations, subscribe to the listservs, or have access to a computer.Future research should incorporate additional sampling techniques such as probability sampling via telephone surveys, or convenience sampling with broader coverage (such as reaching out to individual OST providers through mail or targeting OST professional development conferences and workshops).In addition, the pilot surveys did not explicitly define the term "primary job responsibilities," which left the respondents to interpret its meaning.Future research should include its definition within the survey to limit variation in the interpretation.
Furthermore, while exploratory factor analysis can be a powerful analysis tool to reduce data, its results are most appropriately interpreted with two notes of caution.First, EFA does not identify a unique solution, but rather an optimal solution that minimizes the correlations between each factor.Thus, future samples based on larger, probabilistic designs may generate alternate solutions.Second, this sample was more highly educated than previous estimates of the human services workforce suggest (Light, 2003).This may account for the larger proportion of participants falling into Categories 1-3 (Upper-level and Mid-level Administrators, and Program Staff).In the future, researchers should make a concerted effort to contact larger subsamples of Intermediary Staff and Service Providers (Categories 4 and 5 respectively).Third, dichotomous variables for each job responsibility were utilized in the EFA.The identification of the factor structure relies on linear regression estimation of each variable with each identified factor, thus continuous indicators are most appropriate.Since dichotomous indicators do not fulfill the assumption of a normally distributed variable, results warrant a note of caution (Agresti & Finlay, 1997).Future research should further explore the underlying factor structure using continuous indicators of job responsibilities (such as, "How many days in a typical week do you spend performing each of the following job responsibilities?"), as well as perform confirmatory factor analyses within a structural equation model to provide a more formal test of the proposed factor structure.

Discussion and Suggestions for Implementation
This study presents a new classification system for categorizing OST job characteristics and responsibilities in both program and professional development evaluation.Based on quantitative data from two online surveys combined with qualitative focus group data, we present evidence supporting the utilization of six categories of job titles for describing staff roles in OST research and evaluation: upper-level administrators, mid-level administrators, program staff, intermediary staff, service providers, and "other."Each of these categories are linked with a larger set of unique job responsibilities that can be used to further describe each grouping as well as to assist survey participants in making the appropriate selection.This classification may be a useful resource for researchers who seek to draw comparisons across OST organizations within the US.
When considering these results, researchers and practitioners alike should keep in mind the exploratory nature of both the sampling procedure and analytic framework.Since neither were intended for formal hypothesis testing, there may be limited practical applicability of this classification until future research explores these relationships in a more confirmatory manner.Other: ___________________________________________________ This new classification has been field tested within an evaluation for a national after-school conference and in several professional development workshops for OST staff.The related question includes the following directive for respondents: "I currently spend most of my work week on: (choose ONE answer only)…" Preliminary findings from the evaluations suggest that respondents comply with this alternative system.

Conclusions and Suggestions for Future Research
Overall, the findings extend previous research by providing a new system of gathering demographic information from OST staff.Including either the research-based or "alternative" classification system within a quantitative or qualitative format can successfully and efficiently illustrate the roles of OST workers in the field.Moreover, this can be utilized as a means of comparing workers across diverse workplace settings throughout the country.Future research should further test the accuracy and utility of the research-based classification system using generalizable sampling designs, continuous indicators of job responsibilities, and factor analysis-both exploratory and confirmatory.In the meantime, the alternative classification will likely yield valuable information for OST researchers who are interested in more immediate implementation.
Lastly, professional development practitioners and evaluators may find similar utility from this classification as it allows one to explore how different staff members react to professional development workshops and conferences.Such exploration could push the field of OST professional development further towards achieving its goal of effectively communicating new practices with staff members, and may ultimately contribute to increases in student outcomes as staff are better prepared to function within their programmatic roles.

Table 1
Respondent Demographics from the Second Plot Survey

Table 2
Correlation Matrix, Job Responsibilities in the Second Pilot Survey Interestingly, it yielded very comparable results but with a few important alterations.The first category was split into two different groups: Upper-level Administrators (such as Presidents and CEO's) and Mid-level Administrators (such as Program Directors).The second and third categories were replicated exactly, and a new category emerged: Service Providers.With the revisions noted above, a new classification would be as follows:Category 1 (Upper-Level Administrators): Manage an organization, manage a budget, write grants, or fundraise.Category 2 (Mid-Level Administrators): Manage/Monitor one or more program sites, manage a budget, or supervise paid staff.
For example, Category 3, Program Staff contains substantively different job tasks (e.g., direct service, create/develop programs, supervise volunteer staff, OR provide office support).It may be that two subcategories are subsumed within this single category, which future research using larger surveys may be able to parse out.Further, the distinction between "technical support" identified in Category 3 (Program Staff) and the responsibilities within Categories 4 (Intermediary Staff) and 5 (Service Providers) is not readily apparent.Resources/referrals and licensing support typically are considered 'technical support,' and all may be considered intermediary support provision, so practically these categories are confusing.Until future research can develop a stronger empirical base for this classification utilizing probability sampling, continuous measurement of job responsibilities, and exploratory as well as confirmatory factor analyses, an alternative categorization of the results may be useful for practitioners interested in immediate implementation.This alternative classification makes two changes.First, it reduces the categorical choices to only five options, combining Categories 4 and 5 (Intermediary Staff and Service Providers) into a single category: "Capacity Building," as many of the job responsibilities described reflect activities designed to increase organizational capacity.Second, instead of "Program Staff," the new category is renamed "Direct-Service" and the primary job responsibility is limited to working directly with children/youth.Upper-Level Administration (e.g., Manage an organization, manage a budget, fundraise, and/or work with a governing board.)Mid-LevelAdministration (e.g., Manage/direct one or more program sites.)Direct-Service (e.g., Work directly with children/youth.)Capacity-Building (e.g., Provide professional development, provide technical assistance, monitor programs, evaluate programs, and/or conduct research.)