Paper Summary
Share...

Direct link:

Assessing Interrater Reliability With Multiple Raters: Coefficients to Report in an Educational Context

Mon, April 7, 8:15 to 9:45am, Convention Center, Floor: Terrace Level, Terrace IV

Abstract

Teacher evaluation has been an increasing focus since the inception of Race to the Top. As such, states and school districts around the country have been creating new systems to assess teacher quality. In doing so, the newly created instruments need to be assessed for inter-rater reliability. Past reliability studies have typically provided a percent of overall agreement and some variation of Cohen’s Kappa as evidence of inter-rater reliability. While this is commonplace, Kappa statistics have been shown to suffer from a paradox which causes the estimate to be artificially low. Evaluation data from a school district are used to both illustrate the paradox and to provide support for the inclusion of Gwet’s AC1 statistic into educational reliability studies.

Authors