Impact of Repeater Judges on Intra- and Interjudge Consistency in an Angoff Standard Setting

• In Event: 29.082 - Roundtable Session 9
  In Roundtable Session: 29.082-13 - Improving the Accuracy of Human Rating and Scoring

Abstract

This study explores the impact of serving repeatedly on a standard setting panel on intra- and interjudge consistency. When judges participate in repeated standard settings, does their own performance improve from the standpoint of intrajudge consistency, and does the standard settings they participate in improve from the standpoint of interjudge consistency, error reduction, and dependability increase? Generalizability theory was used to evaluate judge and item x judge variability changes, standard error of the cut score, and dependability index changes in standard settings with repeater judges. It was determined that participating in several standard settings with similar training may not reduce intrajudge variance, while it may improve interjudge agreement to some degree.

Author

• Tanya Longabach, Excelsior College