Out-of-School Technology Programs : Creating Brighter Futures for Youth

Findings from a new, comprehensive evaluation sponsored by the Bill & Melinda Gates Foundation confirm that out-of-school technology programs create brighter futures for youth. Results indicate with high reliability, that these programs can improve academic achievement, build positive self-esteem, and prepare youth for the 21st century workplace. Individual organizations have drawn these conclusions since they began offering youth technology programs; however, the lack of independent multi-year, multi-program evaluations have made it difficult for the field to make concerted programmatic and funding decisions. Findings from this evaluation decisively show that youth technology programs yield significant benefits both, in the short and long-term. This paper offers insights and information relevant to both providers and funders of individual programs, as well as to the advancement of the youth technology field as a whole.


Introduction
Since out-of-school technology programs arrived on the scene in the 1990's, they have been popular with youth.Studies have shown that youth benefit from being able to fine-tune the skills they learn in school, as well as the opportunity to use hardware and software not otherwise available to them.Past research has shown that youth benefit from technology programs in terms of skill acquisition and immediate academic achievement.Studies have also proven that technology programs go a long way to bridging the digital divide, giving low-income students access to technology they need to succeed in the 21st century (Hall & Israel, 2004;Henriquez & Ba, 2000).Until now, however, little to no research was available about how outof-school technology programs impact youth long-term.This article presents findings from the first multi-year, multi-program evaluation that confirms out-of-school technology programs create brighter futures for youth.It outlines specifically how programs benefit youth and offers suggestions for ways to design programs for maximum impact.Findings from this study, sponsored by the Bill & Melinda Gates Foundation and administered by MGS Consulting, Inc., provide evidence for both managers and funders that can be applied to individual youth technology programs and the field as a whole.

The Community Access to Technology (CAT) Program
Many out-of-school technology programs were developed as a response to the ever-widening digital divide, which was leaving low-income youth with limited access to increasingly utilized and useful technology, especially when compared to their middle and high-income counterparts (Office of Educational Research and Improvement, US Department of Education, 1999).As part of the effort to bridge this divide, the Gates Foundation founded the Community Access to Technology Program (CAT) in 1999.CAT's mandate was to help Washington state nonprofit organizations provide local communities greater access to digital technology.The focus of the CAT program was on supporting organizations that help at-risk youth, persons with disabilities, homeless persons, immigrant populations, Native Americans, and rural communities to use technology to make positive changes in their lives and in their communities.From 1999 through 2005, the foundation granted $14 million to 330 organizations and sites in Washington State, resulting in services being provided to over 78,400 people.
In 2005, CAT was transferred to the Center to Bridge the Digital Divide program (CBDD) of Washington State University and was renamed the Communities Connect Network to emphasize the efforts to bring together community technology interests from across the state.Refer to www.communitiesconnect.org for additional information.

Advancing the Field: Rationale for a Multi-Year Evaluation
Extensive evaluation has been done on individual out-of-school programs in general and technology programs in particular.However, little was known about the long-term impact technology programs have on youth participants and the communities in which they live.To fill this void, MGS Consulting began conducting a three-year evaluation of the CAT program in 2003.For each year, the goal of the evaluation was to quantify and gain a better understanding of the individual and societal impact that the CAT grant portfolio had on youth.

Table A
Overall, 885 youth participants (ages 9-19) were surveyed at 36 different program sites throughout Washington State.From this grouping we gathered responses from a subset of 85 youth participants who participated in a survey three times during the 15-month evaluation process (referred to as the "matched set").
Two methods of data collection were used: 1) surveys completed by the youth participants at each site and 2) interviews with key program staff.MGS consultants visited each site for 1-2 days three times over the course of the evaluation period.Youth who participated in the surveys were youth who happened to be at the individual sites during those times.Program staff at each site were specifically asked not to target or invite specific youth to attend at the times of our visits, in order to avoid selection bias on the part of the program leadership.Youth could opt out of participating in the survey, and all responses come from youth that were independently present and willing to participate.

Surveys
MGS Consulting designed the first of a two-part online survey using Survey Monkey.The survey was completed by youth at each of the 36 participating sites on three separate occasions.The first part of the survey asked for data regarding measurable changes in the participant's perception of employability, confidence levels, and grades.The second part of the survey was the Search Institute's Developmental Assets Profile, and consisted of forty questions with a 4-point rating scale.The Search Institute is an independent non-profit organization that seeks to provide leadership, knowledge, and resources to promote healthy children, youth, and communities.More information on the Search Institute can be found at www.searchinstitute.org.According to their resources: "The DAP is a 58 item checklist that takes about 10 minutes to complete and is focused exclusively on assets.It is an individual measure that yields quantitative scores for asset categories and context areas portrayed in a profile format… It is designed to be sensitive to changes in reported assets over time and it is suited to research and program evaluation.MGS staff traveled to each site and personally facilitated survey administration.If a site did not have Internet connectivity, the participants filled out paper copies of the survey and their answers were manually entered into the online form by MGS staff.
There was a four-month (within 100-150 days) period between survey administration at each site.This time period was within the range for validity, according to the DAP guidelines.

Staff Interviews
Interviews were conducted with key personnel to ascertain the key elements necessary for a successful program.The survey included the following topics: • The characteristics of a high-quality technology program • Witnessed or anticipated academic and job readiness outcomes • Social support structures that the staff see as necessary components of a successful tech program Thirty-two key staff members were interviewed in person by MGS staff, one small group discussion was held, and two written surveys were completed.All program staff and key personnel with knowledge of the program (such as program managers or the Executive Director) were invited to participate, and those that did were self-selected.

Results
Hard data from this evaluation effort shows that out-of-school technology programs create brighter futures for youth.Students perform better in school, have increased self-esteem, confidence in their technology skills, and a belief that they will succeed as they move forward.Perhaps the most surprising -and heartening -findings relate to the increase students experience in developmental assets.Each major finding is discussed in turn below.

Developmental Assets
Developmental assets are… " 'developmental vitamins' -positive experiences and qualities identified by the Search Institute that are essential to healthy psychological and social development in childhood and adolescence.These assets have the power to influence young people's developmental trajectories, protect them from a range of negative outcomes, and help them become more productive, caring, and responsible adults," (Search Institute DAPP Users Manual, 2004, p. 2).
Developmental assets are closely related to various behavioral and academic outcomes among youth, as demonstrated in the following table: (DAPP, 2004).Results from this study indicated that youth participation in community technology programs leads to an increase in perceived developmental assets.Further, we discovered that youths' perceived developmental assets increased with time spent in a program.The number of hours a youth participated per week in a technology program was positively correlated with their DAP score.This held true for all ages, genders, and program types.

DAP Score by Time Spent in Program
Increased DAP score leads to a chance of other improvements.The Search Institute divides asset levels into four categories, which we will refer to as "Low" (0-10 assets), "Fair" (11-20 assets), "Good" (21-30), and "Excellent" (31-40 assets).An example of this categorization can be found at http://www.search-institute.org/power-assets.Their research has found that jumping from one category to the next is linked to dramatic increases or decreases in risky and healthy behaviors (Benson, Roehlkepartain, & Sesma, 2004 Roehlkepartain , 2003).The Search Institute notes that " Most young people in the United States -regardless of age, gender, or region in the country, experience too few of the 40 assets," (Search Institute, 2006, p. 4).In order to maximize one's chances in life, scoring at minimum at the low end of the "Excellent" category (having 31 out of 40 assets) is preferred.
Our research shows that participating in out-of-school technology programs was linked with an increase in assets and progression into higher categories of assets.This means that, after participating in youth technology programs, youth have a much greater chance of engaging in healthy behaviors and a greatly decreased risk for engaging in risky or destructive behaviors.By the end of our study, 62% of youth participants who started in the "Low" DAP category moved up to "Fair" or "Good," and 33% moved from "Fair" to the "Good" or "Excellent" category.These participants greatly reduced their risk of engaging in problem behaviors, such as alcohol use, violence, illicit drug use and sexual activity, and increased their likelihood of engaging in positive activities such as succeeding in school and maintaining good health.
As a result of these findings, we conclude that these community youth technology programs greatly increase the chance of the future success of the youth who participate.

Technical Fluency
Technical fluency reflects the participant's ability to understand, explain, and discuss technical tools and concepts.It contributes to what have been called the "21st Century Skills" identified as: Digital Age Literacy, Inventive Thinking, Effective Communication, and High Productivity (NCREL/Metiri Group, 2003).These aptitudes have been identified as critical skills needed to succeed in today's world.Examples include understanding how a computer processes information, describing how email is transmitted, and feeling confident in learning new software applications.When asked about their own abilities, youth participating in these programs report that they are indeed increasing their technical fluency, leading to success in each of these four 21st Century skill sets.

Key Findings
Technical fluency increases with time spent in a program.Fluency and time in program are positively correlated.This means that as youth spent more time in the program, they were increasingly likely to answer "Always" in response to questions regarding their skill and comfort level with technology.
For all participants, males had higher tech fluency scores than females.This difference may be due to issues of confidence rather than skill acquisition.This finding suggests that girls may need programming and support to help build confidence in their own abilities.This finding is consistent with a study of tech fluency skills at Southwestern University: "'The American Freshman: National Norms for Fall 2000" revealed that male and female college students have identical rates of computer use, but the men are twice as likely as the women to have a high opinion of their skills…The majority of males see themselves as having a higher fluency…than do females," (Fass McEuen, 2001, p. 11).
High-touch program participants report higher tech fluency scores than participants in low-touch programs.For all participants, high-touch programs had an average tech fluency score of 2.57, while lowtouch sites had an average score of 2.42.These results indicate that youth need guidance and structure to truly enhance their skills.
High-touch programs are more structured and tend to offer guided projects and classes.Youth are encouraged to learn, create, and work with leaders and their peers.Low-touch programs tend to be less structured and offer more lab time rather than classes.An example of a hightouch site we looked at is the Technology Access Foundation (TAF).TAF has a defined curriculum, attendance requirements for participants, and encourages parental involvement in the enrollment and learning process.Other program sites we reviewed, such as the Boys and Girls Clubs, are low-touch by design.Parents are required to sign a general permission slip, and youth utilize open access labs that don't require attendance or curriculum adherence, as well as areas for homework help and peer interaction.
Technical fluency is higher for teens than preteens.While this result is partly expected, it may also suggest that programs should encourage youth to begin participating in community technology programs when they are preteens, allowing them to participate for multiple years and have adequate time to grow and develop their skills.

Outlook for the Future
Believing you can accomplish something is the first step toward success.This positive mindset is powerful.Research shows that a positive attitude and belief that one will attend college is linked to actual college attendance rates (Carpenter & Fleishman, 1987).In our study, youth who participated in the technology programs believed that they would do better in school, attend college, and get a good job after they finish school.

Key Findings
• The longer the youth participants participated in the technology programs, the more positive their outlook on their future became.Over a 15-month period, youth in the matched set became more optimistic about their school performance: More youth believed they would improve their study skills (60% in the first survey vs. 70% in the third) and graduate high school (72% vs. 76%).• We saw the following changes regarding job outlook and readiness: • Seventy-seven percent of youth new to the program reported that they would get a good job, compared with 90% of youth who had been in a program for more than three years.• After time in a program, more youth said they believed they would get a good job when they were done with school (83% vs. 88%).• More youth in high-touch programs than low-touch programs thought participating would help them go to college (70% vs. 62%).However, more youth in low-touch programs thought participation would help them in school (67% vs. 60% in high touch programs).This implies that high-touch programs focus on preparing for the future while low-touch programs focus on succeeding today.• Preteens have a more positive outlook than teens.As youth age, more is needed programmatically that will help them continue to see their potential.
• In our matched sample, girls showed the greatest gain in outlook.
• Unique skills are acquired in the first year.Youth were asked if the skills they were learning by participating in these programs were skills that could or could not be learned elsewhere.Youth reported that they acquired the most skills that cannot be learned elsewhere in the first year of participation.

Characteristics of Effective Programs
Several individual sites had higher than average scores for at least one of the main indicators (developmental assets, technical fluency, or outlook for the future).Key factors to the success of these sites included: • Highly involved, consistent, and dedicated staff.
• A permanent location that is easily accessible by youth and offers an inviting, comfortable atmosphere.• Going beyond technology for technology's sake, i.e. connect technology to the everyday (and academic) lives of participants to make technology feel relevant and allow participants to see practical and applicable uses for the skills they are learning.• Integrate soft skills, e.g.job interview skills and conflict management.

Developmental Assets
Our findings suggest that different participants have different experiences in individual programs, with females and preteens showing the most improvement.DAP scores for females increased the longer they participated in a program, and scores for preteens increased regardless of how long they were in a program.Interestingly, the two groups who seemed to benefit the most represent the minority of all participants (39% females vs. 61% males; 43% preteens vs. 58% teens).Perhaps programs should revise their marketing to target more females and preteens, since it is with these two groups that they seem to have the most impact.
Or perhaps the current curriculum and/or organization of the technical program is not appropriate.The majority of participants are male and teens, yet these are the precise groups reporting the least improvement in DAP scores.Program leaders should look for any disconnects between who they are serving and how the program functions (including curriculum, leaders, timing of programs, etc).With some marketing and program adjustments, perhaps a more balanced group of males and females, teens and preteens will participate and show gains in DAP scores and begin reversing the national downward trend to reflect the positive trend seen among participants in out-of-school technology programs.

Tech Fluency
As we have seen, on average, tech fluency scores increased over time, indicating that these programs are an effective method for strengthening youths' confidence in their technology skills.The longer youth participate in the programs, the more likely they are to say that statements reflecting technical skills ("I can easily learn new software applications") are always true and the less likely they are to say they are never true.
However, tech fluency skills didn't increase at the same rate for all participating youth, most noticeably for females and youth in low-touch programs.For females, this may be more an issue of confidence rather than skill.Previously mentioned research shows that males and females tend to have the same level of skill but differing amounts of confidence in those skills (Fass McEuen, 2001).Technology programs should focus on increasing the confidence of the females who attend their programs.They must go beyond simply teaching girls skills; they must also give them the confidence to believe in and use their skills in a meaningful way.Increasing the participants' confidence, and thereby their ability to express their true knowledge, is a key aspect of the communication skills associated with 21st Century Skills.
The fact that youth in low-touch programs have lower tech fluency skills indicates that youth need more than just access to improve their skills.High-touch programs are more structured and tend to offer guided projects and classes.Youth are encouraged to learn, create, and work with leaders and their peers.Low-touch programs tend to be less structured and offer more lab time, rather than classes.The results of our study indicate that youth need guidance and structure to truly enhance their skills.Programs that only offer access may indeed provide a safe and positive place for youth to congregate outside school.While such an environment may be valuable in and of itself, it's not the same as actively working to increase technical skills.Programs intended to strengthen the tech fluency skills of their participants should consider incorporating some aspects of high-touch programs (such as structured classes and projects and highly engaged program staff).

Outlook on Future
Over time, youth who participate in these programs believe more is possible for their future and their outlook improves.This result is encouraging for programs aimed at influencing youth in a holistic manner, teaching them life skills and a positive outlook as well as improving their technical skills.
Technology programs can take advantage of their positive effect on participating youth by building a component into their programs focusing on future opportunities.For example, the Tacoma Intel Computer Clubhouse takes participating youth on field trips to local businesses, allows them to job shadow adults working in their field of interest, and encourages them to pursue summer internships.These types of activities offer youth the opportunity to imagine what their futures could look like and encourage them to dream big.

Questions for Future Research
The question remains whether the benefits found in out-of-school technology programs can be generalized to other out-of-school programs, i.e. is it the technology skills themselves that create benefits for youth or do out-of-school programs that focus on other skills and/or activities deliver similar benefits?For instance, would youth involved in sports report similar gains in their developmental assets scores over time?Would those playing in a jazz band view their futures in a more optimistic light the longer they played?These are questions that can and should be answered so that parents, managers, and funders know which investments of time and effort will yield the results they seek.
Another area of potential future research would be looking at elements that make up an effective program.As mentioned earlier, when we looked at the sites that had higher than average scores for at least one of the main indicators (developmental assets, technical fluency, or outlook for the future), we found some key common elements to these sites were dedicated staff, involved parents, and an accessible/inviting environment.Further analysis of the validity of these as indicator elements of successful programs may be warranted, as well as exploring for other leveraged elements and assets.

Conclusion
These findings prove that out-of-school technology programs create brighter futures for youth.Further evaluation should be done to gain a deeper understanding of the value of out-of-school programs in general vs. programs that focus on specific skill acquisition, e.g.technology.As demands on students' out-of-school time escalate, it is important that funders, practitioners, and parents alike understand the ways in which specific activities will impact youths' futures so they are armed with the best possible information on which to base decisions.
Consulting conducted the evaluation from 2003 to 2006.Ten organizations took part, including the Yesler Community Center, Stone Soup, the 4H, and the Washington State Boys and Girls Club Alliance (See Table A for full list).All had received grants as part of the CAT Program.
Girls Clubs of America / Washington State Boys & Girls Club Alliance Search Institute 40 developmental assets framework, a strengths-based approach for promoting the healthy development of children and youth.The framework defines the relationships, opportunities, skills, and values children and adolescents need to thrive," (Search Institute Developmental Assets Profile Preliminary Users Manual, January 2004).