Establishing and Monitoring Interrater Reliability of Classroom Observations in a Large-Scale Randomized Controlled Trial

• In Event: Classroom Observation Protocols in Multiple Learning Settings

Abstract

We demonstrate how inter-rater reliability (IRR) can be established and monitored in a large-scale randomized controlled trial (RCT) study, using two different classroom observation instruments (Transitional Bilingual Observation Protocol [TBOP], and Teacher Observation Record [TOR]). We closely examined the training, coding procedures, and calculations of inter-rater reliability (IRR) across treatment and control conditions on these two multi-domain-response-rater fidelity of implementation (FOI) instruments. The observations were conducted virtually at the beginning, middle, and end of 2015-2016 school year in Grade 1 bilingual classrooms across seven school districts in Texas. We propose an inter-rater process flow to guide the steps in (a) establishing initial IRR, (b) continuously monitoring IRR, and (c) re-establishing IRR should it fall below the cut-off value.

Authors

• Cindy Lynn Guerrero, Texas A&M University

• Shifang Tang, Texas A&M University - College Station

• Shuqiong Lin, Texas A&M University - College Station

• Tamara Lopez, Texas A&M University - College Station

• Fuhui Tong, Texas A&M University - College Station

• Beverly J. Irby, Texas A&M University - College Station

• Rafael Lara-Alecio, Texas A&M University