Retention of Basic Science Information by Senior Medical Students and Graduates on the United States Medical Licensing Examination

• In Event: 60.044 - The Best Predictor of the Future May Be the Past, but How Accurate Are the Predictions? Estimating Professional Competence Using Diverse Methodologies

Abstract

Perspective: Over the past 40 years, NBME studies of retention of basic science information consistently have shown that performance declines as trainees advance. In the past 20 years, basic science curricula at many schools shifted from discipline-based teaching toward integrated instruction where students learn basic science in clinical context. Concurrently, basic science assessment changed: at many schools and on USMLE Step 1, there is increased emphasis on applying basic science knowledge in clinical context, rather than simply recalling basic science facts.

Objective: This study investigated whether changes in basic science education and assessment affected patterns of retention, extending previous work to include Step 3.

Methods and Data Sources: 244 basic science items from 2006-07 administrations of Step 1 were included as unscored items on 2008-09 Step 2 Clinical Knowledge (CK) and 2009-10 Step 3 test forms. Item responses of 15,000+ first-time examinees from US/Canadian medical schools were analyzed to identify item characteristics influencing shifts in performance. Items were classified according to Stem Format (Patient vignette, Experimental vignette, Non-vignette) and Committee of Origin (Anatomy, Behavioral Sciences, Pathology, etc). Mean Step 1, Step 2 CK, and Step 3 item difficulties (p-values) were calculated as percentages of first-time examinees from US/Canadian schools answering correctly. The study qualified for exempt status from the American Institutes for Research Institutional Review Board.

Results: Overall, mean item difficulty on Step 1 was 75%; for Step 2 CK and Step 3, this value declined to 71% and 64%, respectively. As shown below, declines in mean item difficulties were observed for all test committees of origin except Behavioral Sciences, with the largest decline for Biochemistry & Genetics on Step 3. Declines also were seen for each stem format; these were largest for Experimental vignettes on Step 3 and smallest for Patient vignettes on Step 2 CK.

Step 1 Step 2 CK Step 3 Step 2 CK – Step 1 Step 3 – Step 1
OVERALL RESULTS 75% 71% 64% -4% -11%
Format of Item Stem
Non-vignette 73% 61% 55% -12% -18%
Experimental vignette 72% 60% 53% -12% -19%
Patient vignette 76% 73% 66% -2% -9%
Test Committee of Origin
Biochemistry & Genetics 71% 56% 40% -16% -31%
Microbiology 79% 70% 63% -9% -16%
Anatomy 76% 69% 63% -7% -13%
Pharmacology 75% 68% 63% -7% -12%
Physiology 72% 67% 57% -5% -15%
Pathology 76% 77% 71% 1% -4%
Behavioral Sciences 80% 90% 90% 10% 10%

Significance: Performance shifts from Step 1 to Step 2 CK and Step 3 were similar to those observed in previous NBME studies, though the magnitude of overall differences was larger. Results are disappointing: one might expect that growing use of integrated basic science curricula would be reflected in improved retention of basic science, particularly for items framed in clinical context. Explanations for decline in performance may include 1) lack of reinforcement of basic science content during clinical training, 2) masking of effects of integrated curricula by students’ extensive preparation for Step 1, and 3) exclusive use of multiple-choice question formats to assess application of basic science knowledge.

Author

• David B. Swanson, American Board for Medical Specialities