Rater Connections and the Detection of Bias in Performance Assessment

• In Event: AERA Roundtable Session 10
  In Roundtable Session: Missing Data in Measurement

Abstract

In many performance assessments, one or two raters scores each performance, resulting in a sparse rating design. Relatively limited observations of each rater pose challenges for identifying raters who exhibit idiosyncrasies such as differential rater functioning (DRF; i.e., rater bias). When raters who exhibit DRF are directly connected to other raters who exhibit DRF, there is limited information with which to detect this effect. A simulation is used to systematically examine how the nature of connections among raters in sparse designs impacts DRF indices. Results indicate that the nature of connections among DRF raters has a substantial impact on DRF indices, and that monitoring ratings and variable rater assignment to student performances can improve DRF detection.

Author

• Stefanie A. Wind, University of Alabama