Evaluation of Implementation and Sustainability of the Ohio Network of Education Transformation: Cross-Model Findings

• In Event: Evaluation of a Multilevel Supported System Working to Implement Diverse Promising Educational Reform Programs

Abstract

Purpose
The Ohio Network of Education Transformation (ONET), created by the Ohio Department of Education (ODE) with Race to the Top (RttT) funds, aimed to improve lower functioning schools across three years of implementation of one of five innovative whole-school reform models. This presentation highlights evaluation findings for four of these models - AVID, ISSN, Early College, New Tech - across two academic years in two schools each (8 schools total), using quantitative and qualitative methods. Topics discussed include implementation successes and barriers, sustainability facilitators and hindrances, student and school outcomes, and recommendations gleaned from lessons learned.

Theoretical Framework
Reviewing almost a half century of studies, Nunnery (1989) found that schools were more likely to obtain desired student gains if they partnered with an external service provider. Payne (2008) reported the development of generations of local reform efforts has produced “So much reform, so little change.” Findings like these have generated interest in using external reforms as a promising path to school improvement. Ohio is one of dozens of states experimenting with supporting diverse external reform designs as paths to improved student outcomes. A near-universal finding has been that variance in implementation levels is very large. Both Payne (2008) and Supovitz and Weinbaum (2008) have produced detailed descriptions of “implementation gaps” leading to reform failure. All of this points to the importance of understanding not just how specific reforms are intended to work, when described in the abstract, but the range of ways they perform in the often very challenging environments of diverse schools. Herein we draw upon findings across four different models to better inform recommendations for future school reform successes.

Methods and Data Sources
This evaluation used a mixed-methods multiple case study design (Yin,2014). Data collected consisted mostly of interview and/ or focus group data across all of the sites including observational data collection methods. The sample included multiple key stakeholders across 8 schools (principals, assistant principals, curriculum developers and model experts, and teacher).

Findings
There was tremendous variance in levels of implementation across schools and models with 17 identified factors contributing to whether models were sustained one year after ONET funding ended. Seven of the eight schools sustained their models to some degree. Issues of leadership, funding, professional development, and planning for transitions were among the most often observed in determining levels of implementation. Student outcomes – measured via Ohio’s School Report Card– did not evidence dramatic improvement for several plausible reasons (e.g., implementation, time needed to observe change, match of indicators and model goals). Qualitative reports by principals and teachers, however, suggest that the models had value, especially teachers’ and principals’ skills development and new appreciations for students’ potentials.

Scholarly Significance
Six major recommendations were gleaned from lessons learned: caution against sole reliance on state level outcomes measures; need for more planning time prior to implementation; facilitate schools focusing on one reform at a time compared to multiple reforms; provide tangible supports (e.g., funded support personnel/ experts); insist on transition plans; mandate on sustainability plans.

Authors

• Samuel C. Stringfield, University of Cincinnati

• Vicki L. Plano Clark, University of Cincinnati

• Jacinda K. Dariotis, University of Cincinnati

• Ann M. Allen, The Ohio State University

• Jessica west

• Jennifer Kessa Roberts, The Ohio State University

• Kathleen Carr, Strategic Research Group

• Audra Morrison