More Than Accuracy and Reliability: Principals' Learning of Consequential Observation Skills

• In Event: 29.046 - Administrative Sense-Making in Assessments of Teaching

Abstract

With a few exceptions (e.g., Cash, et al, 2012), research on the current generation of K-12 observations has conceptualized the work of observation primarily from a measurement perspective (e.g., Bell et al, 2012; BMGF, 2012; Grossman et al, 2010). By this we mean the studies have conceptualized raters as observers who receive training that disciplines their view of teaching to match the observation protocol. Observers are then evaluated on the degree to which their scores are accurate and consistent with master observers’ scores on the same lessons. The background of the observer has been largely irrelevant; and the relationship between the teacher and the observer has not received attention.
As observation protocols begin to be used by principals for the sorting of teachers and the improvement of teaching, it is important to understand the degree to which previous research applies to practice. This comparative case study draws on three perspectives to analyze the research question: How do principals in the Los Angeles Unified School District understand and use the district’s new teacher evaluation observation protocol - the Teaching and Learning Framework (TLF)? The three perspectives—organizational, socio-cultural, and cognitive—conceptualize the creation of observation scores as work that is influenced by issues such as the supervisory relationship, the goals of the principal, and the degree to which the TLF matches the principal’s view of good instruction. These are issues generally unexplored in the measurement literature on observations.
The analyses in this paper draw on data from a study of principal learning during the first years of district-wide TLF implementation. For each of the 42 focus observers with whom we collaborated for two years, we watched training and collected data from pre, mid and post training questionnaires, five interviews, three implementation questionnaires, and two think aloud exercise designed to see how observers understood the TLF.
Preliminary analyses paint a complex picture of sense-making and instrument use. Principals’ descriptions of the main goal in conducting observations ranged from “making sure students are learning” to “helping teachers get better at what they want to get better at”. Notably, no principals saw their work as fundamentally about creating accurate and reliable scores. Instead, principals saw the TLF as part of the work they do to improve instruction in their buildings.
Principals did not see the TLF as different from their view of instruction. However, they also did not yet demonstrate familiarity and competency working with TLF scoring rubrics. Some principals immersed themselves in the TLF and nominated specific aspects of the TLF they needed to learn better. Interestingly, principals who used the instrument less frequently nominated more general needs. Finally, principals saw their relationships with teachers being critical to the observations resulting in teaching improvements.
These preliminary results suggest that in order for observation protocols to support the improvement of teaching, researchers and trainers will need to understand how to bring together a view that privileges score accuracy and reliability with principals’ views that privilege relationships and multiple views of how to improve teaching.

Authors

• Courtney A. Bell, ETS

• Nathan Jones, Boston University

• Jennifer M. Lewis, Wayne State University

• Yi Qi, Educational Testing Service

• David Kirui

• Monica McLeod, Detroit Public Schools