Investigating Rater Bias in Objective Structured Clinical Examinations Under a Many-Facet Rasch Model Framework

• In Event: Constructing Learner Assessments

Abstract

We conducted an analysis of an objective structured clinical examination (OSCE) program using a Many-Facet Rasch Model (MFRM) framework to identify and adjust for rater bias. We found that bias in the ratings provided by standardized patients tended to be minimal, but additional training may be recommended to alleviate unpredicted judgements. Items tended to be somewhat easy; however, given that this is a mastery test, care should be taken to ensure that the scoring models align with the purpose of the assessment. Furthermore, MFRM requires a partially crossed rating plan and we found two disconnected subsets. Addressing the connection between subsets will improve the quality of the measures in future analyses.

Authors

• Michael R Peabody, IXL Learning
  Presenting Author

• Shannon O. Sampson, University of Kentucky
  Presenting Author

• Kelly D. Bradley, University of Kentucky
  Presenting Author

• Jagriti Chadha, University of Kentucky
  Non-Presenting Author

• Alan Hall, University of Kentucky
  Non-Presenting Author

• Helen Garces, University of Kentucky
  Non-Presenting Author

• Andres Ayoob, University of Kentucky
  Non-Presenting Author