Comparing Cumulative Standard Errors of Levine and Tucker Linear Equating for a High-Stakes Clinical Performance Examination

• In Event: Equating Scores on Large-Scale Medical Performance Assessments: Practical Issues and Solutions

Abstract

Background

Our organization uses Levine observed score linear equating to link scores over time for a high-stakes objective structured clinical examination (OSCE) under nonequivalent-groups with anchor design. Best practice indicates periodic review, as such we are evaluating the current method and explore other options such as Tucker linear equating (Clarke, et al. (2017).
The accuracy of linking/equating OSCEs may be affected by systematic or random errors. Systematic errors can be introduced by factors such as violations of statistical assumptions. Tucker and Levine methods rely on different assumptions, some of which may not be strictly met in our situation. Random errors occur when samples from a population are used to estimate equating relationships, and consequently, the results are subject to sampling errors. Random errors can be quantified by standard errors (SEs) of equating. The SEs of one single equating link are usually not large enough to cause concern. However, after a chain of equating links, SEs can accumulate and affect score accuracy. Many studies evaluated SEs of a single equating link for multiple-choice tests. Few were in the context of equating clinical performance exams with multiple links.

Purpose
Tian, et al. (2017) examined SEs of chained equating for linking OSCEs across years using Levine method and found that SEs accumulated as the number of links increased. The present study extended this investigation to compare Levine and Tucker methods in terms of cumulative SEs with multiple links. A bootstrap method was used to estimate SEs using real examinee data.
Method

We used data from seven test forms with sample size ranging from 180 to 498. Figure 1 depicts the design for four chains of three-link equating, with each linking one recent form (X3A, X3B, X3C, X3D) to Form Y (base) through Forms X2 and X1.

To estimate SEs for a chain, e.g., X3A→X2 →X1 →Y, the following steps were performed separately using Levine and Tucker methods.
1. Draw a bootstrap sample from each form
2. Equate X3A to X2
3. Equate X2 to X1
4. Derive the slope and intercept using the equating relationships in Steps 2-3, and convert X3A scores to X1 scale
5. Equate X1 to Y
6. Derive the slope and intercept using the equating relationships in Steps 4-5, and convert X3A scores to Y scale
7. Repeat above steps 10,000 times
8. Compute the SEs for one-, two- and three-link equating across 10,000 simulations at Steps 2, 4 and 6

Results
Similar patterns for Levine and Tucker:
• SEs accumulated as the number of links increased
• SE accumulation was smaller near the mean, larger towards the extremes
• Accumulated SEs were smaller than SEM
• In general, Tucker produced smaller SEs but the differences between the two methods were extremely small

Educational Significance
This study supported the use of either Levine or Tucker method for linking OSCEs, but highlighted the need to address the issue that SEs may accumulate to an extent that could cause scale drifting, affect score and decision accuracy as the number of links keeps increasing.
 

Authors

• Fang Tian, Medical Council of Canada

• Andrea Gotzmann, Medical Council of Canada

• Sirius Qin, Medical Council of Canada

• Liane N. Patsula, Medical Council of Canada

• André F. De Champlain, Medical Council of Canada