K-12 School Accountability in New York City: Using Multiple Measures of School Quality and Effectiveness

• In Event: Designing and Refining: The Development and Maintenance of Accountability Metrics in New York City

Abstract

This paper explores the methods the New York City Department of Education (NYCDOE) uses to evaluate New York City K-12 schools and publicly report on their performance. The approach is based on the Framework for Great Schools and includes both student achievement measures like test scores and graduation outcomes and non-academic measures from survey results and school visits. This accountability approach focuses on identifying areas for improvement and support and highlighting school characteristics that drive student achievement.

The Framework elements are based on evidence showing organizational features of schools beyond student achievement are associated with substantial improvement in schools. In a longitudinal research study, the Chicago Consortium of Schools identified five features: Effective Leaders, Collaborative Teachers, Involved Families, Supportive Environment, and Ambitious Instruction, and an embedded trust measure across the features (Bryk, 2010). These five elements, plus a sixth Trust element, were adopted by the NYCDOE in the 2014-15 school year (NYCDOE, 2015). Framework results are publicly shared through School Quality Report products, including the Snapshot, Guide, and Dashboard.

At the core of the Framework for Great Schools is the Student Achievement Rating, which is based on how schools perform relative to targets set in advance. Targets are set using an internally-developed Comparison Group method, similar to nearest neighbor and propensity score matching approaches. In this methodology, each student at a school is matched to fifty similar students at other schools based on demographic characteristics and incoming test scores. The performance of this group of similar students is used to generate targets for schools.

The Framework measures use student-level NYCDOE administrative data, including demographic information, test scores, graduation results, grades, and postsecondary outcomes. Data sources also include results from the New York City School Survey and Quality Review. By incorporating multiple measures of school performance, the Framework elements differentiate between the various components that make up a high-performing school. Each year NYCDOE sets metric targets in advance, making methods more transparent and facilitating school leaders’ decision making. The Comparison Group method controls for students’ incoming characteristics so that schools serving high need students are not unfairly penalized.

Testing for fairness of the Framework, we find that both Framework and Student Achievement ratings have low correlation with school-level aggregates of economic need, language status, and special education status. Controlling for incoming factors through the Comparison Group method does not significantly affect validity of the measures, as evidenced by high positive correlation between Student Achievement ratings and other measures of school quality. We also find the measures are stable year-over-year. Between 2015 and 2016, 63% of schools stayed at the same rating, ~20% moved up one rating, ~16% moved down one rating, and less than 1% moved more than one rating. Finally, the Framework elements and Student Achievement are interconnected to one another. Similar to findings from the five Chicago elements, we observe a relationship where schools with strong Framework element ratings in one year have strong Student Achievement ratings in the following year.

Authors

• Kelly E. Dietz, New York City Department of Education

• Erin Dooley, New York City Department of Education

• Jimmy Kaiser, The New York City Department of Education

• Andrew Lee, New York City Department of Education