Exploring Decision Consistency Across Rating Designs in Rater-Mediated Assessments

• In Event: 29.082 - Roundtable Session 9
  In Roundtable Session: 29.082-13 - Improving the Accuracy of Human Rating and Scoring

Abstract

Although complete rating designs, in which each rater scores every student, are applied frequently in research, practical constraints limit their use in operational performance assessments. Instead, rating designs are frequently applied in which raters are randomly assigned to score student performances; these designs reflect a range of connectivity among facets in the assessment system. Using simulated and operational performance assessment data, this study explores the impact of differences in connectivity within rating designs on decision consistency when adjustments are made for differences in rater severity. Preliminary results suggest that connectivity is an important issue that should be included in the psychometric evaluation of rater-mediated assessments to warrant the validity, reliability and fairness of the judgments obtained from raters.

Authors

• Stefanie Anne Wind, The University of Alabama - Tuscaloosa

• Pey Shin Ooi, The University of Adelaide

• George Engelhard, University of Georgia