Estimating Stringency Parameters for Standardized Patients in a Large-Scale Performance Assessment

• In Event: Equating Scores on Large-Scale Medical Performance Assessments: Practical Issues and Solutions

Abstract

Theoretical Framework
This study focuses on issues of accuracy in estimating equating parameters in the USMLE Step 2 Clinical Skills examination, or Step 2 CS®, a large-scale standardized patient (SP) examination and part of the licensure examination sequence for physicians in the US. We consider here the components scored by SPs using checklists and rating scales. Each examinee is scored based on their performance in multiple clinical scenarios.

Step 2 CS scores are equated to adjust for differences in SP and rater stringency, as well as task difficulty. In this proposal, we describe the problem as it applies to the examination components scored by SP. Since SPs are trained to provide ratings on one clinical scenario (or case), a combination of the case and SP is the unit of analysis for calibration [1, 2, 3]. The case-SP combination comprises both the effect of the difficulty of the case and the SP’s stringency. Estimation of case-SP parameters (calibration) is accomplished by using the ordinary least squares method and a univariate ANOVA model. For this model, the standard errors are described by standard formulae using mean squared errors [4].

Methods
The calibrations are run on case-SP score data about every 8-10 weeks. The frequency is dictated by the number of available SP ratings (450 case-SP combinations, about 60,000 ratings) and bank update processes schedule. Yet, the reality of the scoring process is such that parameter estimates are required for new SPs who are newly trained or matched to a case and were not included in the annual large ANOVA calibration. To prevent a delay in the scoring process, the NBME developed a process to calibrate case-SPs outside of large ANOVA-based calibrations. The single-case calibration procedure uses only aggregate statistics on the most recent ANOVA parameters and existing ratings for the particular case-sp requiring an estimate.
The single-case procedure calculates a difficulty parameter for the new case-SP R as follows:
,
where
is the grand mean of parameter estimates from the current bank of case-SP parameter estimates; S denotes a sample of people who were scored by the new case-SP. Accordingly, we named

the case-SP offset. It shows how much more stringent or lenient the new case-SP is as compared to the rest of the case-SPs within the site; and
the examinee offset. It shows how much less or more able is the sample of examinees who saw the particular case-SP.

Results
The preliminary results showed that parameter estimates obtained using the single-case procedure are close to the ones ANOVA would provide, given the number of observations used is high enough (45 or more). In the simulation study we expand the study to push the number of ratings to the limit required by ANOVA and compare accuracy with ANOVA parameters.

Educational Significance
The particularly attractive feature of this operational procedure is that it estimates the new case-SP parameter using response data from a limited number of encounters with a new case-SP and results of the regular ANOVA.

Authors

• Irina Grabovsky, National Board of Medical Examiners

• Amanda Clauser, National Board of Medical Examiners