The Role of External Evaluation: Promoting Continuous Quality Improvement as a New Accountability

• In Event: Education Inside the Juvenile Justice System: Leveraging Data, Research, and Evidence to Drive Improvement in a System Under Tension

Abstract

Since 2021, the Massachusetts Department of Youth Services (DYS) has contracted with an external research and evaluation firm to evaluate the quality of services provided to youth in DYS custody and care. Initially, the project was structured by a monitoring and accountability framework that intended to “assure the quality” of educational services provided to youth in custody. As the complexity and the challenges of the juvenile justice education system became clear, and trust was established between CES, the external evaluator, and DYS, questions of monitoring and accountability shifted towards continuous quality improvement. Instead of asking: “Is the Collaborative for Educational Services (CES) doing what is laid out in their contract with DYS?” we investigated questions like: “Under what conditions are CES’ educational services a) being implemented with fidelity and b) what progress is being made towards short- and mid-term outcomes?

Simultaneously, CES’ and DYS’ joint commitment to using accelerated learning to address young people’s lagging skills set a clear direction and boundaries for our team’s work. Using a mixed-methods approach that included surveys of youth in custody, classroom observations, and analyses of extant data, our team has engaged in providing ongoing, data-informed feedback to multi-stakeholder groups. Informed by the “Plan, Do, Study, Act cycle” that undergirds a continuous quality improvement approach to evaluation, the purpose of this feedback was to enable responsive adaptation of the accelerated learning initiative. In a recent and first-of-its-kind gathering of leaders from DYS and CES, the external evaluation team facilitated activities that invited grappling with pivotal issues in the accelerated learning initiative and the juvenile justice education space more broadly. For instance, we asked CES to respond to the fact that the decision to roll out the accelerated learning initiative in treatment (committed) programs only - because those programs provided opportunities to co-plan and co-teach with special education teachers - meant that youth in detention would not have access to the standard of services. In another moment, CES challenged DYS’ policies that restrict sending DYS transcripts to detained students’ district schools upon discharge - a practice intended to protect privacy that has the unintended consequences of burdening youth and families with the task of requesting and sharing transcripts across agencies, and of making students feel that their education inside DYS doesn’t count towards anything.

Previous panelists in this symposium have illuminated the lack of nationwide and even within-state accountability systems for juvenile justice education programs. While it does not fill the gap, this case study sheds light on an alternative approach to using data and research to drive improvement in a fraught policy system. Specifically, we highlight an approach grounded in a different model of accountability: cross-agency collaboration, trusting relationships, and honest reflection on mixed-methods data. This approach itself may give rise to better data. Inspired by emergent questions, the evaluation team, DYS, and CES are presently problem-solving the challenges of administering a pre-post survey to youth as they enter and leave custody, a challenge made more pronounced by the different policies and practices of discharging detained youth, sometimes without notice.

Author

• Meg Caven, Education Development Center, Inc.