Comparison of Automated Scoring Algorithms Developed for a Patient Management Assessment by Independent Groups of Practicing Physicians

• In Event: Assessment Issues in Competency-Based Education in the Professions

Abstract

Perspective: The development of automated scoring guidelines for complex performance assessments is an iterative process. Research investigating the impact of using different groups of content experts to generate rules that inform automatic scoring algorithms is fairly uncommon.
Objective: The purpose of the present study was to compare the consistency of rule-based scores for a computer-based patient management assessment across scoring algorithms developed by different groups of content experts.
Methods and Data Sources: The current study compares scoring algorithms created and defined by three independent groups of content experts for the computer-based case simulations (CCS) component of the Step 3 United States Medical Licensing Examination (USMLE). Computer-based case simulations require the examinee to manage a virtual patient in a simulated patient care environment. The full examination comprises 12 simulated clinical cases where examinees freely navigate the interface to manage the virtual patient. In this study, scoring algorithms and action categories were developed for a common set of four simulated clinical cases. Two of the three independent groups (Group 1 and Group 3) had three content experts while one group (Group 2) had four content experts. Each content expert rated 30 cases after the group’s rules were developed. Analyses were completed at the case level using the same group of examinees (N = 1,585).
Results: The table below contains descriptive statistics (Mean (SD)) for automated scores for 1,585 examinees based on the scoring rules developed by the three groups of experts. Results indicate that scores varied across cases and groups; average scores ranged from 3.67 (1.72) for Group 3 for Case 2, to a high of 7.12 (1.84) for Group 1 for the same case. Overall, the scoring rules developed by Group 3 were most stringent across cases and of the four cases, case 3 seemed to be easiest.
Case Group 1 Group 2 Group 3
1 7.12 (1.84) 6.86 (1.55) 3.84 (0.61)
2 4.41 (2.24) 5.78 (1.98) 3.67 (1.72)
3 5.77 (1.60) 6.58 (1.91) 6.34 (2.36)
4 4.44 (2.15) 5.69 (2.50) 4.47 (2.39)

Shown below are the correlations between scores from each of the three independent groups of content experts for the sample of examinees available for scoring. Correlations for the full sample of examinees ranged between 0.48 and 0.89.
Case Group 1 and
Group 2 Group 1 and
Group 3 Group 2 and
Group 3
1 0.67 0.67 0.74
2 0.61 0.77 0.56
3 0.68 0.48 0.65
4 0.77 0.89 0.79
Mean 0.68 0.70 0.69

The table below reports the correlations between the ratings produced by each content expert in the three independent groups and the scores for the sample of 30 performances. Average rating score correlations were fairly consistent. Correlations for rated samples ranged between 0.80 and 0.97.
Case Rating and
Group 1 Rating and
Group 2 Rating and
Group 3
1 0.94 0.97 0.81
2 0.91 0.80 0.84
3 0.86 0.94 0.95
4 0.89 0.87 0.86
Mean 0.90 0.90 0.87

Significance: Development of scoring guidelines used to automatically score complex performance assessments is group specific and should be considered as a source of measurement error.

Author

• Ulana A. Luciw-Dubas, National Board of Medical Examiners