Supporting Data Use for Continuous Improvement: Lessons From California

• In Event: 66.051 - Testing Continuous Improvement in Practice: Learning Locally to Improve at Scale

Abstract

Though the use of data in the education system is not a novel practice, data use for continuous improvement reflects a recent and critical change in mindset. In this paper, we provide a framework for how to use data to support continuous improvement and discuss current data use structures in California. By detailing the strengths and weaknesses of approaches in California, we provide guidance to scholars and policymakers about how to better support schools and districts in using data for improvement.

In our framework, we show that data use for continuous improvement, with its adaptive and iterative nature, differs from data use for other purposes, including accountability (Berwick, James, & Coye, 2003). Data use for improvement entails a cycle of collecting and interpreting data, constructing ideas on potential solutions, making appropriate modifications, and monitoring and researching whether changes resulted in improvement (Mandinach & Honey, 2008). In its basic form, this iterative cycle transforms data into usable knowledge and thus makes it actionable. For data to be utilized effectively for improvement, there must be a deliberate structure around its use, with various actors aligned in both goals and process (Park, Hironaka, Carver, & Nordstrum, 2013). The specific data necessary to drive improvement also depend on the phase of the improvement process (The Victorian Quality Council, 2008) and the user (Annenberg Institute for School Reform et al., 2014).

Using California as a case study, we discuss the state’s policy context and the current state of data use based on document review and interviews conducted in the summer of 2017 with 41 leaders from state and regional education agencies, school districts, technical assistance providers, education advocacy organizations, and education associations. To analyze interviews, we used a multiple case study design (Yin, 2017; Miles, Huberman, & Saldaña, 2013). We find that while the state’s new data reporting tools have been welcomed as an improvement over previous accountability measurement systems, policymakers and educators across the state agree that state data systems are limited as a resource for districts to review, analyze, and make decisions to improve the system. Specifically, the state data systems suffer from the following limitations: a) the static nature of the displays makes it difficult to study trends over time; b) the “one school at a time” displays make it difficult to compare across or between systems; c) limited breadth of data and lack of comparisons across metrics make it difficult to build a holistic, integrated understanding of performance for a school; and d) the inadequate frequency of data collection limits the ability of schools to track real-time impact as they make changes to improve. Districts instead need interactive data systems that facilitate the benchmarking of student performance against other schools, with structured network learning activities that enhance their ability to use data to drive continuous system improvement, which they access through other means. Since not all schools and districts have access to this additional data support, there are large implications for equity and improvement at scale.

Authors

• Heather J. Hough, Stanford University

• Erika Byun, Stanford University

• Laura S. Mulfinger, Claremont Graduate University