The Complexity of Rater Training

• In Event: Validity Arguments for K–12 Teacher Observational Measures: Issues and Future Directions

Abstract

Rater training typically includes rater agreement or “training reliability” at the conclusion of the training phase. Training reliability is usually reported in publications with evidence such as: “Once pairs of raters have achieved at least 80% exact-point agreement in the field (or using videotaped lessons from a project’s dataset), lessons may be observed/scored by an individual rater” (Boston, Bostic, Lesseig & Sherman, 2015, p.5). The previous description of a training reliability criterion is fairly typical in both length and content—about 1 sentence, describing a general threshold. In a review of 10 years of research published in the Journal for Research in Mathematics Education and Reading Research Quarterly, only 10 of the 17 studies using quantitative classroom observational tools included a description of training reliability (Wilhelm, Gillespie Rouse, & Jones, 2018). In this presentation, the speakers will describe the complexity of decisions and interpretations of the statement and related implications for data reliability. We also make a case for more precise attention to this important phase of the coding process.

The ME observational tool is a set of classroom observation rubrics that were designed to attend to aspects of high-quality mathematics instruction while focusing on specific practices that reflect the most current research on how to support more equitable participation and student access in inquiry-oriented learning environments. Data for this methodological analysis come from our larger research project focused on the validation of the rubrics for particular uses.

In this analysis, we draw on data in our first round of coding involving three raters, all doctoral students in education, in preparation for generalizability and decision study. We set out to quickly train the three raters, deem their codes reliable on a separate set of videos (at the 80% exact agreement threshold), and move on to coding the study’s data. However, we recognized the importance of trusting the data that was produced, so we began to rethink our criteria for training reliability. We will use training reliability data to describe different decision points and implications for rater agreement. For example, 80% exact agreement taken most simply could just mean that for 8 out of the 10 rubrics the rater assigned codes that matched the codes assigned by a master rater. But, for the other two rubrics, the rater did not match the master rater’s codes. In contrast, we identified that we wanted codes on every rubric to be as consistent as possible. Further, we wanted to ensure that coders were in agreement for the range of scores possible for each of the rubrics, a hard thing to do when it is a new tool and there is not a large bank of previously scored videos to use to round out the training.

With little guidance for rater reliability with teacher observational measures, common practice is to do what others have done (Wilhelm, Gillespie Rouse & Jones, 2018). We aim to describe some of the complexity of using teacher observational measures and offer some alternative approaches to assessing training reliability.

Authors

• Anne Garrison Wilhelm, Southern Methodist University

• Jonee Wilson, North Carolina State University

• Elizabeth Lynn Adams, Southern Methodist University

• Temple A. Walkowiak, North Carolina State University

• Anna Thorp, North Carolina State University