Indicators of Inter-Rater Reliability for Classroom Observation Instruments as Fidelity of Implementation in Large-Scale Randomized Controlled Trials

• In Event: 44.085 - Poster Session 9
  In Poster Session: 44.085-10 - Large Scale Assessment SIG Poster Session

Abstract

Using two different classroom observation instruments with frequency (Transitional Bilingual Observation Protocol [TBOP]) or Likert rating data (Teacher Observation Record [TOR]) in a large-scale randomized controlled trial (RCT), we examined the inter-rater reliability (IRR) of observations across treatment and control conditions. These two multi-domain-response-rater instruments were used as measures of fidelity. The observations were conducted virtually at the beginning, middle, and end of 2015-2016 school year in 116 Grade 1 bilingual classrooms across seven school districts in Texas. We propose the use of two indicators of IRR (intra-class correlation coefficient and Gwet’s AC1) that are most appropriate for continuous and categorical data, respectively

Authors

• Fuhui Tong, Texas A&M University - College Station

• Beverly J. Irby, Texas A&M University - College Station

• Shifang Tang, Texas A&M University - College Station

• Shuqiong Lin, Texas A&M University - College Station

• Cindy Lynn Guerrero, Texas A&M University

• Rafael Lara-Alecio, Texas A&M University

• Tamara Lopez, Texas A&M University - College Station