Detecting Rater Biases in Sparse Rater-Mediated Assessment Networks

• In Event: Rasch Modeling Methodologies

Abstract

Practical constraints in rater-mediated assessments limit the availability of complete data. Instead, most scoring procedures include one or two ratings for each performance, with overlapping performances across raters to facilitate model estimation. These incomplete scoring designs present challenges for detecting differential rater functioning (DRF). The purpose of this study is to explore methods for detecting DRF in rater-mediated assessments that involve sparse rating designs based on “linking sets” of examinee responses made up of multiple-choice items or a common set of performances, and to compare the results to a design with systematic connections based on overlapping performances across raters. Preliminary results indicate improved sensitivity and specificity for DRF indices with linking sets compared to overlapping performances.

Authors

• Stefanie A. Wind, The University of Alabama - Tuscaloosa

• Yuan Ge, University of Alabama