Impact on Scoring When Rater Effects Are Ignored in Performance Assessments

• In Event: Equating Scores on Large-Scale Medical Performance Assessments: Practical Issues and Solutions

Abstract

Theoretical Framework
Performance assessments in medicine commonly assess clinical skills based on a clinical scenario where a standardized patient (SP) interacts with a candidate scored by a SP or physician examiner (PE). Swanson, Clauser and Case (1999) describe common data collection models when considering scoring, equating, and test security. Typically, educational or testing programs collect data where each SP/PE interact with a set of candidates on a station and each combination of SP/PE is different across stations (i.e., a nested design). This design is advantageous for testing a large group of candidates, but it does not account for rater effects in estimating scores, as some overlap across SP/PE is required to disentangle these effects. Ideally, we want to account for these rater effects (i.e. lenient and stringent raters) from the candidate’s total score in a scoring model. In this study, we simulated a data collection design that is ‘connected’ to assess the impact of rater effects and four scoring models on candidates’ scores.

Methods
One test form from a high-stake medical licensing examination was used to estimate the 293 item parameters across 12 stations using the partial credit model (Masters, 1982). These parameters were used to simulated data for four types of rater conditions: (no rater effect, six lenient, six stringent, mix of six lenient and six stringent) and two levels of score changes (small/moderate). Four scoring models were assessed in this study, based on two conditions: (1) using item scores/station scores and; (2) estimating the rater effect or ignoring it. This resulted in a 7 x 4 simulation study with 28 conditions.

A total of 10 replications were generated for each condition (500 for full paper) where each rater scored 250 candidates on two different stations. To fully connect this design, we simulated one ‘super rater’ with no rater effect who rated all 500 candidates and stations.
For all conditions, the many-facet Rasch model (Linacre, 1989) was used to estimate ability based on item difficulty and rater effects. The following four outcome measures were averaged across replications for each of the conditions: (1) pass rates, (2) classification changes, (3) Decision consistency and accuracy, and (4) Root mean squared estimates for item and theta estimates.

Results
Preliminary results indicate that rater effects should be estimated in high-stake contexts as ability estimates were either lower or higher than expected depending on the direction of the rater effect simulated. For example, for a large increase/decrease in half the raters, the difference between observed and expected pass rates ranged from 1.8%-2.2% (when rater effects were estimated), to 2.9%-9.6%, when the rater effects were ignored. This suggests that some passing candidates should be failing and vice-versa, based on raters that are either too stringent or lenient.

Educational Significance
The results of this study show that for high-stakes performance assessments, scoring models and data collection designs should incorporate rater estimates. In future studies, we plan on evaluating test form equating, observed score methods, and other data collection designs.

Authors

• Andrea Gotzmann, Medical Council of Canada

• Sirius Qin, Medical Council of Canada

• Maxim Morin, Université de Montréal

• André F. De Champlain, Medical Council of Canada