Using Multilevel Item Response Theory to Explain Differential Item Functioning on a Medical Maintenance of Certification Exam

• In Event: Roundtable Session 9
  In Roundtable Session: Factors Contributing to Performance Differences of Practicing Physicians in Internal Medicine

Abstract

One way to help ensure that irrelevant variance is not present in high-stakes secure exams is the assessment of different item functioning (DIF). DIF occurs when two or more groups of examinees of the same proficiency level have significantly different probabilities of answering an exam item correctly. Several studies on medical certification and Maintenance of Certification (MOC) exams found that few items exhibited DIF, with approximately equal number of items favoring either group (Dugosh, Song, & Ostrem, 2013; Zhang & Cubbellotti, 2015). However, a recent study found that IMGs took noticeably longer to complete certification and MOC exams (Zhang & Cubbellotti, 2015), suggesting that IMGs may face some challenges worthy of investigation.
The purpose of this study was to examine factors that make items differentially more difficult for IMGs (or USMGs). A hierarchical generalized linear modeling framework allows specifying a series of multilevel item response theory (ML-IRT) models to detect and explain DIF using item and person characteristics (De Boeck & Wilson, 2004). The advantage of this methodology is that by applying the models to a large certifying board’s 2014 MOC exam data (Table 1) we were able to:
• Obtain item difficulties and person ability measures;
• Assess impact and test for DIF across IMGs and USMGs;
• Examine item characteristics (e.g., item length) that affect item difficulty;
• Test for the differential effects of item characteristics on item difficulty for IMGs and USMGs;
• Examine how person characteristics (e.g., age) moderate the effects of item characteristics on item difficulty for IMGs and USMGs.
Detailed interpretation of coefficients will be provided in the full paper. Similar to Cid, Pastor, and Goodman (2010), we used a logit difference of 0.5 or greater to flag items for content review. Out of 170 items, 20 exhibited DIF of .5 logits or more, with most items falling slightly above .5, and only one item above 1 (Figure 1). Among these items, half favored IMGs and the other half, USMGs. Across all examinees, items containing images and judgment items were relatively easier; synthesis items and longer items were more difficult (Table 2). Moreover, image and judgment items were slightly more difficult for IMGs; however, this effect was not statistically significant (Table 3). In addition, as examinees’ age increased, synthesis items became easier for USMGs, whereas longer items became easier for IMGs (Table 4). Finally, controlling for testing time, synthesis items were more difficult for IMGs; the negative effect of item length was exacerbated for USMGs who took longer to complete the test (Table 5).
The findings of this study have important implications for test developers and assessment practitioners and should be examined further. Specifically, this work sheds light on the relationship of item length, cognitive tasks, and the use of media with test performance. In addition, person-level variables beyond age and testing time warrant further investigation to better understand the different effects of item features on IMGs and USMGs, respectively.

Authors

• Bozhidar M. Bashkov, American Board of Internal Medicine

• Stephen Cubbellotti, American Board of Internal Medicine

• Yanling Zhang, American Board of Internal Medicine