The Use of Visual Representations of Scoring Bias in Rater Training

• In Event: Roundtable Session 12
  In Roundtable Session: Development and Validation of School-Based Assessments

Abstract

In this study we examined the use of visual representations of types of scoring bias to train raters. Initial evidence is mixed about whether the quality of raters’ scores improved when their training incorporated visual representations of scoring bias versus all verbal descriptions of bias. Two of the three treatment groups (i.e., raters trained with visual representations of bias) displayed closer agreement with validation scores than did the controls; whereas, two of the control groups had higher interrater reliability than the treatment groups. The potential for the utility of visual representations of bias is reflected in that a majority of raters in the treatment group successfully recalled more of the types of bias that were presented visually rather than verbally.

Authors

• Robert L. Johnson, University of South Carolina

• Min Zhu, University of South Carolina - Columbia

• Brandon Loudermilk, Lenoir-Rhyne University

• Ashlee A. Lewis, University of South Carolina

• Xiaofang Jae, University of South Carolina