Building Collective Capacity for Program Quality Improvement: Boston Beyond’s Certified Observer Network

Emily Dodge


This article describes Boston Beyond’s effort to develop a network of out-of-school time program partner staff trained in implementing a program quality observation tool. Participant survey (n = 63) and interview (n = 4) feedback demonstrate that the network is meeting its goals of advancing participants’ professional development, positively impacting organizations, and creating a system-level model of peer-to-peer program quality improvement. Areas of improvement are identified for each goal. Questions of sustainability, planned network improvements, and considerations for others seeking to establish similar networks are discussed.


observation; program quality; program improvement

Full Text:



Akiva, T., & Yohalem, N. (2006). Quality systems: Lessons from early efforts to disseminate the Youth PQA. Washington, DC: Forum for Youth Investment. Retrieved from

Duffett, A., & Johnson, J. (2004). All work and no play? Listening to what kids and parents really want from out-of-school time. New York, NY: Public Agenda and the Wallace Foundation.

Durlak, J. A., Weissberg, R. P., & Pachan, M. K. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal Community Psychology, 45, 294-309.

Eccles, J., & Gootman, J. (Eds.). (2002). Community programs to promote youth development. Washington, DC: National Academy Press.

Hilberg, R. S., Waxman, H. C., & Tharp, R. G. (2003). Introduction: Purposes and perspectives on classroom observation research. In H. C. Waxman, R. G. Tharp, & R. S. Hilberg (Eds.), Observational research in U.S. classrooms: New approaches for understanding cultural and linguistic diversity (pp. 1-20). Cambridge: Cambridge University Press.

Larson, R., Rickman, A. N., Gibbons, C. M., & Walker, K. C. (2009). Practitioner expertise: Creating quality within the daily tumble of events in youth settings. New Directions for Youth Development, 121, 71-88. 10.1002/yd.297

Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-school time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76, 275-313.

Mahoney, J. L., Harris, A. L., & Eccles, J. S. (2006). Organized activity participation, positive youth development, and the over-scheduling hypothesis. Social Policy Report: Giving Child and Youth Development Knowledge Away, 20, 3-32.

Pianta, R. C. (2010). Standardized classroom observations from pre-K to third grade: A mechanism for improving quality classroom experiences during the P–3 years.

Pianta, R., & Hamre, B. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38, 109-119.

Shulman, L. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15, 4-14.

Smith, C., & Akiva, T. (2008). Quality accountability: Improving fidelity of broad developmentally focused interventions. In H. Yoshikawa & B. Shinn (Eds.), Transforming Social Settings: Towards Positive Youth Development. New York: Oxford University Press.

Tracy, A., Charmaraman, L., Ceder, I., Richer, A., & Surr, W. (2016). Measuring program quality: Evidence of the scientific validity of the Assessment of Program Practices Tool. Afterschool Matters, 24, 3-11.

Tracy, A., Surr, W., & Richer, A. (2012). The Assessment of Afterschool Program Practices Tool (APT): Findings from the APT validation study. Wellesley, MA: National Institute on Out-of-School Time.

Wilson-Ahlstrom, A., & Yohalem, N. (2007). Building quality improvement systems in the youth-serving sector: Lessons from three emerging efforts. Washington, DC: Forum for Youth Investment.


Copyright (c) 2020 Emily Dodge

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.