A Comparison of Instructional Coaches’ and External Raters’ Observational Scores

• In Event: Validity Arguments for K–12 Teacher Observational Measures: Issues and Future Directions

Abstract

The STEM Teacher Observational Protocol (STEM TOP) was designed to support systematic coaching efforts within the context of a middle school STEM initiative called the STEM Academy for Science Teachers and Leaders. Initial evidence of validity supported the internal structure of the tool for measuring two dimensions of instruction including: (a) STEM instruction and (b) classroom management and discipline (Authors, 2019, 2020a). The purpose of this proposal is to evaluate evidence of rater reliability.

We investigated 25 teachers’ mean scores across time on the STEM TOP during the 2018-19 school year. The STEM TOP was completed by instructional coaches, who worked with teachers as a part of their participation in the STEM Academy. Four coaches completed the STEM TOP at seven timepoints throughout the school year, which we termed cycles. STEM Academy coaches engaged in in-depth reliability training including training reliability requiring that 80% of the scores matched the master score exactly or within one and an additional 20% double observations with fellow coaches throughout the school year. Based on the double observations, scores across raters did not differ substantially when comparing the STEM TOP coaches’ scores to one another (Author, 2020b). As such, evidence of rater reliability based on double observations suggested that raters’ scores were consistently applied across time.

The STEM Academy team hypothesized that teachers’ scores on the STEM TOP would increase across time in STEM Academy teachers’ classrooms. However, steady increases based on the STEM TOP were not observed. We shared these results with instructional coaches, who believed that we could attribute the lack of consistent change to coaches’ expectations for teachers increasing following coaching sessions with teachers. Following the coaching sessions, coaches reported expecting teachers to implement recommended practices. If coaches did not observe those recommended practices, coaches reported scoring teachers “more harshly,” which is evidence of rater drift (Pianta, 2012). This type of rater drift was not captured in the rater reliability analysis based on double observations because coaches scored lessons with consistently increasing expectations across time. We examined rater drift further by comparing external raters and coaches’ scores for the same lessons (n=46). We are currently in the process of analyzing these data, which will allow for better understanding of this type of rater drift within the context of a coaching intervention.

Authors

• Elizabeth Lynn Adams, Southern Methodist University

• Leanne R. Ketterlin-Geller, Southern Methodist University