Balancing Multiple Priorities in Teacher Evaluation: Responsibly Designing Student Growth Measures

• In Event: Designing and Refining: The Development and Maintenance of Accountability Metrics in New York City

Abstract

State education laws governing teacher evaluation often specify a general structure for the system, but leave a portion of the details to be decided and implemented at the local district-level. Districts then construct an approach that balances the law, their priorities, and their context. The initial design is not the end of the story either; additional work is necessary to maintain, and even improve, the system.

The goal of this paper is to illustrate how the New York City Department of Education (NYCDOE) handled this situation and has responsibly constructed, evaluated, and adjusted the student growth measures used in its teacher evaluation system. It will start with the introduction of the guiding principles that steered the design of the evaluation system: fairness, feasibility, instructional viability, developmental support, reliability and validity, school-level autonomy, and transparency. How those principles were translated into measurable standards by which student growth metrics could be designed will be the main focus. Attention will also be paid to the use of the guiding principles to define priorities for areas of improvement since the first year of implementation. A recurring theme will be the need to balance guiding principles that are sometimes at odds with one another.

Various results and analyses from the past 4 years (2014-2017) of for-stakes implementation will be presented in order to explore how well the metrics meet the stated guiding principles. These analyses will be both quantitative and qualitative, reflecting the mixed nature of the guiding principles themselves. Analytics will include outcome distributions, correlations to student population characteristics (most of which have a magnitude less than 0.1, demonstrating neutrality and fairness), and tests of reliability (e.g., 70% of teachers are assigned the same rating on the same assessment 2 years in a row) and validity (e.g., correlations with other growth metrics ranging from 0.84-0.92), both at the student- and teacher-level.

The importance of producing fair and equitable results across a multitude of dimensions will be a particular focus throughout the presentation. In a school district as large and diverse as New York City, balancing and accounting for many types of fairness at once can be a complex process. The paper will use that lens to portray the sometimes intricate calculus NYCDOE used to responsibly make decisions about their student growth metrics.

While some features and outcomes of the framework will be unique to NYCDOE’s context, the strategies used to identify and balance priorities will be transferable to any district involved in similar work.

Authors

• Pamela Wong, New York City Department of Education

• Joseph D. Jensen, The New York City Department of Education

• Christina Kim, New York City Department of Education

• Jordan R. Munn, The New York City Department of Education