Quantification and Agnosticism in Instructional Observation Systems

• In Event: AERA Roundtable Session 5
  In Roundtable Session: Classroom Observation Manuals as Measures of Teaching Quality: Possibilities and Pitfalls

Abstract

Objective:
The objective of this paper is to highlight the fundamental limitations of existing, global teacher observation protocols and outline the guiding principles of alternative, fine-grained systems. This paper builds on Praetorius & Charalpmbous’ (2018) and Kelly et al.’s (2020) analysis of measurement properties of global observation systems, focusing on the basic conceptual features of alternative approaches to instructional observation rather than realized measurement properties. While global instructional observation systems, have multiple productive uses, current systems really are limited by how they conceptualize teacher observation.

Perspectives:
An essential feature of many teacher observation protocols is their global approach to measuring instruction, where trained observers provide a summary evaluation of multiple domains of instruction scored over an interval of time. I begin by noting the basic strengths of such an approach. First, many existing protocols are remarkably comprehensive. While any given protocol may have a special emphasis, they are generally quite comprehensive of a large set of domains of instructional practice. Second, global protocols cover all instruction and are not restricted to particular methods/modes of instruction (lecture, question and answer, seatwork) or classroom activity structure (e.g. whole-class vs. small group or individualized). Third, they are well-rooted in the educational sciences of best practices, scoring instruction on a meaningful continuum of effective practice. Taken together, these features mean that global protocols are well designed, in theory, to provide basic accountability of effective practice, as well as to document an individual teachers’ weak points, if such within-teacher variation exists.

Findings:
The inherent focus on a continuum of effective practice, that is, the strong valencing of instructional practice as effective or ineffective, precludes more careful inquiry into teachers’ curricular and pedagogical emphases, trade-offs in time use, and instructional practices where competing theoretical perspectives preclude an a-priori judgement of best practice. Stated differently, in focusing solely on a continuum of effective practice, existing global protocols lack the agnostic quality that would allow a program of research to uncover, empirically, the relationship between instructional practice and outcomes. Drawing on theories of teacher learning and professional decision-making, I argue further that global protocols do not encourage professional educators to make their own decisions on what instructional improvements to prioritize, or encourage instructional experimentation.

Significance:
Turning to uses of aggregate data from systems of teacher observations, I argue that this focus on a continuum of effective practice, along with the basic rough, qualitative nature of lesson scoring in global protocols, means they are not well-designed to assess variation in opportunity to learn, or to document large-scale changes in instructional emphases over-time. To illustrate, consider that many educators today are concerned with whether the cultural content of literature and other curriculum is representative of changing societies, but global protocols don’t measure that content, and certainly not with a precision to be able to detect changes over time. Overall, by focusing solely on a continuum of effective practice, global protocols fail to capture both what is being taught, but also any agnostic dimensions of how instruction is carried out.

Author

• Sean P. Kelly, University of Pittsburgh