Pilot to Policy: Reconsidering the Reliability of Observation-Based Ratings From New Mexico's Statewide Teacher Evaluation System

• In Event: Hilton Roundtable Session 13
  In Roundtable Session: Measurement Issues in the Implementation and Evaluation of Accountability Policy

Abstract

Classroom observation instruments feature prominently in teacher evaluation systems nationwide. While classroom observation enjoys strong support among teachers and school administrators, information about the reliability of observation-based indicators is often collected in low-stakes research settings, including small-scale pilot studies. However, evidence about score reliability collected in low-stakes research contexts may not translate into high-stakes evaluation contexts. This study examines the reliability of observation-based ratings collected in an operational statewide teacher evaluation system in New Mexico. We find that scores based on two observations are reliable, but results suggest that decisions about whether reliability is adequate for intended score uses should be informed by tolerance for error and misclassification. Policy implications are discussed and directions for future research are proposed.

Author

• Jonathan Schweig, The RAND Corporation