Evaluating and Adjusting for Rater Effects in a Standard Setting Using the Angoff Method

• In Event: Poster Session 14
  In Poster Session: Division I Poster Session

Abstract

The validity of standard setting depends on the quality of judgments. This study examines the use of a psychometric rater model to evaluate the quality of judgments provided by expert raters using the Angoff method. A psychometric rater model was used to estimate the precision and severity for each expert rater. Estimates from the model identified raters who were deviant; expert raters differed significantly in their judgments for items that were more relevant to the content than items that were less relevant. Results from this study present an innovative application of psychometric rater models that can provide new and important perspectives to standard setting in the health professions education.

Authors

• Yoon Soo Park, University of Illinois at Chicago

• Kuan Xing, University of Illinois at Chicago