Measuring the Impact of Improvement Research in Out-of-School Contexts

• In Event: The Successes and Challenges of Measurement in School and Out-of-School Improvement Research

Abstract

Out-of-school time (OST) youth development programs are critical for nurturing youth thriving and advancing equity (National Research Council and Institute of Medicine, 2002). Youth development programs often serve as a safe haven or “second home” for young people to build trusting relationships, discover who they are, gain abilities to shape their own lives, and learn how to interact with and contribute to the world around them (Hershberg et al., 2014; Kalish et al., 2010). Despite their importance, OST programs struggle to recruit and retain youth development professionals due to a range of structural barriers such as inadequate funding, lack of advancement opportunities, and underappreciation from the general public (Borden et al., 2021; Noam & Bernstein-Yamashiro, 2013). Communities of Practice (CoP) have been increasingly utilized to improve retention and build youth development professionals’ individual and organizational capacities (Dingyloudi, F. et al., 2019). When CoPs and Improvement Science approaches are coupled together, they create a space for youth development professionals to continually make organizational improvements while in community with other like-minded individuals.

The current paper describes how nineteen youth development organizations in an urban city are using Improvement Science approaches to become relationship-rich organizations. In particular, we will highlight a multi-year, multi-cohort CoP centered around building youth development professionals' capacity to build developmental relationships, foster racial equity, and craft implementation plans using the Annie E. Casey, Results Count Framework (Annie E. Casey Foundation, n.d.). More specifically, this paper will describe the surveys, and interviews used to measure and track organizational improvement efforts. In particular, presenters will share how administering rapid feedback surveys allowed us to get continuous feedback on participant learning and implementation progress, and how value creation interviews allowed us to understand the impact of membership in the CoP and progress with implementation efforts.

Through analyzing participants' value creation interviews, we determined that most youth development professionals in the CoP were making progress towards becoming relationship-rich organizations; however, in some cases, they lacked the critical CEO buy-in necessary for integrating relationship-building into the fabric of the organizational mission and structures. Nevertheless, despite the barriers they faced, participants shared that begin able to break down their implementation plan into smaller targeted Plan-Do-Study-Act (PDSA) cycles, allowed them to focus on specific initiatives such as improving staff professional development. Additionally, a review of participants' rapid feedback surveys highlighted that after attending CoP sessions, participants were prompted to evaluate their existing organization policies and understand the importance of collecting actionable data. Additional supports requested through the rapid feedback survey included needing additional tools as they worked to implement their change-ideas. Overall, regularly administering rapid feedback surveys allowed us to understand participants progress, and incorporate additional relevant resources, training, and discussions of problems of practice during CoP sessions.

Overall, the paper's results highlight the importance of measuring and tracking the progress of youth development professionals' improvement efforts. Additionally, results demonstrate how value-creation interviews can be utilized to measure and understand the impact of both a CoP and participants' implementation efforts.

Authors

• Miray Seward, American Camp Association

• Ta-yang Hsieh, Search Institute