Lesson-Specific Estimates of Internal Reliability: Improving Information Quality and User Trust in Automated Teacher Observation Systems

• In Event: Automated Classroom Observation With Machine Perception

Abstract

Objectives and Theoretical Framework. Teacher observation protocols have several strengths, including that they are lesson specific rather than referencing teachers’ generic practice. Yet, they also have limitations, which limit their utility in research and applied settings (Kelly et al., 2020). In contrast, fine-grained methods of observation utilizing automated tools can potentially overcome these limitations. In particular, here we focus on an important attribute of automated methods that may differ fundamentally from human observation; the internal reliability of automated methods can theoretically be assessed for individual sessions. In contrast, in human observation systems, the reliability of the measurement process is known only in aggregate, and only for a specific group of raters who share common expertise and training. This paper investigates this possibility empirically; How well can the internal reliability of automated lesson observations be predicted from lesson features themselves?
Data and Methods. This analysis used both real and simulated data. A total of 127 observations of English language arts classrooms were conducted in three secondary schools in Pennsylvania (Jensen et al., 2020; Stone et al., 2019). However, the data consist of very high-quality audio where the class engages with disciplinary content interactively. All classroom sessions contained a substantial quantity of teacher discourse during interactive instruction with few interruptions. As a result, this corpus of real-world data may not approximate the quality of measurement future users of such systems might expect. Thus, we also analyzed simulated data, constructed by repeated sampling of the real-world data to create more than 10,000 lesson snapshots with more pronounced variation in input features.
Results. We first considered potential input features that might affect measurement reliability including the predicted value of the discourse feature itself, the quality of automatic speech recognition (ASR) prediction, and the quantity of certain discourse features. We then built regression models to predict, formatively, the likely unreliability in a new classroom observation with given input features. To illustrate model fit, we present predicted values of imprecision (the difference between the automated estimate of teaching quality and a gold standard human estimate) contrasting observations with strong and weak input features. We then apply this method to generate importance weights in examining the distribution and covariance structure of classroom discourse practices in both real and simulated data. When data is collected under controlled research settings, and in high-achieving classrooms, results that account for differential measurement reliability do not differ substantially from naïve estimates because the reliability itself is only so variable. In contrast, results from simulated data suggest that in less controlled applied use settings, where the measurement quality and nature of instruction are more highly variable, identifying unreliable estimates can substantially improve automated teacher observation systems.
Significance. Lesson-specific estimates of reliability are critical to successful use of automated teacher observation systems in professional development and other applied settings—they can be used to remove unreliable feedback to teachers, improving teacher trust and the use-value of observation systems. Even if observations are rarely flagged, the capability of so doing provides a safe-guard to users.

Authors

• Sean P. Kelly, University of Pittsburgh

• Patrick Donnelly, California State University - Chico

• Emily Jensen, University of Colorado - Boulder

• Sidney K. D'Mello, University of Colorado - Boulder