Improving How the New York City Department of Education Assesses School Quality

• In Event: Designing and Refining: The Development and Maintenance of Accountability Metrics in New York City

Abstract

Purpose
For over a decade, the New York City Department of Education (NYCDOE) has used the Quality Review to evaluate schools’ instructional and organizational practices with school improvement planning by providing targeted feedback. Since its inception in 2005, we have continuously refined our methodology toward more rigorous data collection and adjusted our approach to reflect the needs of schools and the system.

Theoretical framework
The earliest assessment of school quality was drawn from successful school evaluation systems, including those used in the United Kingdom and Hong Kong, and comprised of a school visit culminating in a report that includes what the school does well and what it needs to improve on. These main components are still reflected in the Quality Review process today, while the measures and data collection have gradually improved.

Methods
We designed a rubric in 2009 with 20 Quality Indicators that was rooted in best practices across the country and informed by Richard Elmore’s theory of the instructional core. Evaluators conducted two-day school visits to collect evidence and rate each indicator on a four-point scale.

Two years later, we redesigned the rubric to reduce the Quality Indicators from 20 to 10 and grouped them into three categories: Instructional Core, School Culture, and Systems for Improvement.

Prior to 2014, some schools received reviews several years apart resulting in gaps in our system’s data. In 2014 and 2015, we committed to evaluating every school, which required reducing school visits to one day and rating five Quality Indicators.

In 2015, we conducted a process evaluation to improve our methods. We considered the following questions: Which indicators would provide the best feedback to schools and support the accountability system? What is the optimal duration for a school visit? How frequently should schools receive reviews?

Data sources
We conducted interviews and focus groups with stakeholders including policy makers, evaluators, data analysts, superintendents, principals, and school staff, asking them to consider the three questions and reference the Quality Review Rubric. Additionally, we analyzed historical Quality Review data and principal survey feedback.
 
Quality Review Rubric Overview
See Image 1 at end of document.

Results
Principals believed 10 indicators were critical for comprehensive feedback. Further, rating more indicators strengthened our accountability system by providing additional measures for the NYCDOE’s School Quality Reports, in which the Quality Review ratings are a key data source.

Satisfaction with the duration of the Quality Review dropped when it shifted to one day with the majority of principals and evaluators asking for more time. Two-day visits gave evaluators sufficient time to gather evidence and allowed us to formally introduce elements that strengthened our data collection.

We also found that schools benefited from regular feedback to fuel their learning and improvement and our accountability system needed consistent data regarding school quality. Beginning in 2016, we started a three-year cycle of reviews.

Scholarly significance
Evaluations that are meant to serve the needs of schools require necessary adjustments over time to improve the system.

Authors

• Adam Breier, New York City Department of Education

• Elizabeth Guinan, New York City Department of Education

• Yvonne Wang, New York City Department of Education