Principals and the Validity of Observation Scores in High-Stakes Teacher Evaluation

• In Event: Exploring the Implementation and Effects of Various Levers in Multiple-Measure Teacher Evaluation

Abstract

With a few exceptions (e.g., Cash, et al, 2012), policy and existing research on current K-12 observation protocols conceptualizes the work of observation primarily from a measurement perspective (e.g., Bell et al, 2012; BMGF, 2012; Grossman et al, 2013). Policy presumes administrators are raters, trained to set aside their biases and be rigorous and dispassionate while carrying out the specified number of observations and processed defined by the district. Research investigates issues such as certification rates and the degree to which administrators’ scores are accurate and consistent with master observers’ scores on the same lessons.
While the measurement view of teacher evaluation is pervasive and compelling, it largely ignores the organizational context in which administrators act. For example, administrators are often selected for their management skills, not the degree to which they understand or can rate instruction. Further, administrators are principally accountable for school (not teacher) outcomes which are shaped by many reforms and policies, each requiring attention.
This study departs from this measurement view and conceptualizes the work of observing as having three overlapping aspects. Administrators coordinate cognitive (e.g., taking unbiased notes, assigning scores), supervisory (e.g., giving helpful feedback that improves teaching, establishing trust), and leadership (e.g., connecting observations to other building initiatives and the school vision) aspects of teacher evaluation in order to create valid observation scores. We bring these overlapping aspects together to analyze the research question: How do principals understand and use their district’s new teacher evaluation observation protocol?
Drawing on data from 2012-2014 of the Los Angeles Unified School District’s new teacher evaluation system, we use mixed-methods to analyze administrators’ understanding and use of the observation protocol, the Teaching and Learning Framework (TLF). There are three sets of data, 1) pre, mid and post training questionnaires and TLF certification data that comes from roughly 1,200 administrators; 2) in-depth interview, think aloud, and bimonthly survey data for each of the 42 focus administrators with whom we collaborated for two years; and 3) researcher’s field notes from trainings and school visits.
Analyses suggest administrators held some understanding of the TLF, with 89% of administrators certified after training. Exact agreement with master raters on a 4-point scale ranged from 42 to 69%, depending on the domain. Administrators reported learning a lot from their training and analyses document that their note-taking skills improved over time.
In contrast to the measurement view, in practice, administrators used the TLF in unstandardized and complex ways. They set different types of goals for teachers in the same building, focused overwhelmingly on improvement goals rather than evaluation goals, developed professional relationships with teachers as a means to improve teaching practices, and conducted different numbers and types of observations depending on the specific needs of the teacher. Notably, no principals saw their work as fundamentally about creating accurate and reliable scores and viewed the TLF as one part of the work they do to improve instruction in their buildings. These results will be discussed in the context of prevailing views of validity.

Authors

• Courtney A. Bell, ETS

• Nathan Jones, Boston University

• Jennifer M. Lewis, Wayne State University

• Yi Qi, Educational Testing Service

• Leslie Stickler, ETS