Inter-Rater Reliability Estimator Accuracy and Double-Rating Percentages: A Monte Carlo Investigation

• In Event: 30.039 - Exploring Reliability and Validity Measures in Student Assessment

Abstract

Interrater reliability coefficients are often reported in performance assessment as a measure of rating quality. Common estimators of interrater reliability include Pearson product-moment correlation coefficients, Spearman rank-order correlations, polychoric correlation coefficients and generalizability coefficient. This research used Monte Carlo methods to draw samples to examine accuracy of each estimator of inter-rater reliability affected by the initial reliability, the total number of examinees completing a performance task, scale categories, and percentage of double-scored papers. The results showed that with small percentage of double-rated papers (<15%), all estimators had more estimation bias compared to high percentage of double-rated papers. Although each estimator produced an estimate close to the initial reliability, polychoric correlations provided the closest estimates across all conditions.

Authors

• Zhaoxia Guo, University of South Carolina - Columbia

• Ning Jiang, University of South Carolina - Columbia

• Robert L. Johnson, University of South Carolina

• Yin Burgess, University of South Carolina