Refining and Raising the Reliability of Classroom Observations

• In Event: Roundtable Session 50
  In Roundtable Session: Advancing the Science of Classroom Observation: New Applications and Directions From the Field

Abstract

Objectives: In an effort to identify ways to enhance reliability, we focused on two questions: 1) can paired observations increase inter-rater reliability, and 2) how does the number of items and observations influence reliability of classroom observation scores?
Perspective/Theoretical Framework: Enhancing inter-rater reliability is intended to ensure consistency in the feedback provided to UTC’s participants regarding the quality of their teaching practice in specific areas, while enhancing reliability of the overall score is critical to support the use of these scores in making high-stakes decisions regarding whether participants continue in the program. Even when observers are well trained, a single observation conducted by a single observer is a fairly unreliable estimate of a teacher’s practice (Ho & Kane, 2013).
Methods: We attended several of the paired observation sessions and documented how the pairs resolved discrepancies. We also compared inter-rater reliability from the calibration exercises that took place before and after the paired observations. To address the second research question, we conducted a generalizability study to assess multiple elements of the observational system, which inter-rater agreement measures cannot do (Hill, Charambolous, & Kraft, 2012). In a subsequent decision study, we used the estimates from the generalizability study to assess the impact of number of observations and indicators on reliability of observation scores.
Data Sources/Evidence: Data sources included field notes from the paired observations and the coaches’ discussions of ratings, as well as the data on inter-rater reliability from calibration exercises from 9 coaches before and after the paired observations. Post-intervention inter-rater reliability averaged about 11 percentage points higher than pre-intervention inter-rater reliability, and gains were greater for coaches with low reliability prior to participating in the paired observations (Figure 1.1).
For the generalizability and decision studies, the analytic sample consisted of 40 first-year teachers for whom we had at least three observations during SY 2013-14 in which all 19 indicators were rated. The results of the decision study indicated that adding observations and items produces diminishing returns to reliability. Reliability improves most markedly (from .70 to .81 at 10 items) when increasing from 1 to 2 observations. Adding a 3rd observation increases reliability to .85, and a 4th results in .87. Similarly, when we increase the number of items from 2 to 4 (at 2 observations), reliability increases from .63 to .73, but reliability only increases from .79 to .81 when we increase the number of items from 8 to 10 (Figure 1.2).
Scholarly Significance: As data from classroom observations often play a key role in teacher evaluations, ensuring the accuracy of observation scores is essential. We found that participating in paired observations yielded promising improvements in inter-rater reliability and was viewed by coaches as an effective form of professional development. In addition, this study provides empirical evidence to inform decisions regarding the number of items rated and lessons observed needed to achieve desired reliability of observation scores.

Authors

• Cara Jackson, Urban Teachers

• Andrew Dean Ho, Harvard University