Comparative Values of Medical School Assessments for Predicting Internship Performance

• In Event: 60.044 - The Best Predictor of the Future May Be the Past, but How Accurate Are the Predictions? Estimating Professional Competence Using Diverse Methodologies

Abstract

Perspective: Assessments of student learning outcomes help educators understand the effects of their teaching and curricula, as well as provide guidelines for student remediation and educational quality improvement. A variety of undergraduate assessments also are used in the decision-making process for graduate admission. Although studies show the individual value of medical school assessments in predicting residency performance, these findings are not consistent. It is even less clear about their relative contributions when compared simultaneously.

Objective: We compare the contributions of several major academic assessments conducted at various intervals during medical school to the prediction of clinical performance in the first year of residency as rated by program directors.

Methods and Data Sources: Two-hundred-fifty-six (66%) graduates of a medical school were rated by their internship supervisors in residency program director surveys from 2009 and 2011. Multiple regression analyses were used to examine the predictive value on internship ratings of several undergraduate assessments including the United States Medical Licensing Examination (USMLE) Step 1 examination, USMLE Step 2 Clinical Knowledge examination, National Board of Medical Examiners (NBME) medicine subject exam, inpatient internal medicine (IM) clerkship performance ratings, and an 8-station Objective Structured Clinical Examination (OSCE) called Clinical Performance Examination (CPX). A multivariate analysis of covariance (MANCOVA) was conducted to compare the differences among three internship performance groups (Low, Medium, and High) in these measures, using Medical College Admission Test (MCAT) scores as a covariate. This study was deemed exempt from review by UCLA's Institutional Review Board.

Results: All clinical year assessments significantly (absolute values of β ranging from .17 to .25, p < .05) contributed to the prediction of internship performance ratings, although the NBME medicine subject examination contributed in a negative way and only 15% of the variance in the ratings (R2 = .15, F (5,246) = 8.32, p < .001) was explained by the predictors. When the stepwise method was used, only inpatient IM clerkship ratings (β = .21, t = 3.18, p < .01) and CPX scores (β = .19, t = 2.93, p < .01) contributed significantly to the prediction of internship performance. After controlling for differences in MCAT scores, the results showed significant group variations in the undergraduate measures (F (10,488) = 2.91, p = .001, partial eta2 = .06). Follow-up analyses revealed significant (p < .01) group differences in the Inpatient IM clerkship and CPX assessments.

Significance: Results demonstrate that performance in clinical settings are stronger predictors of internship performance than knowledge test scores, and that basic science knowledge measured by Step 1 does not predict internship performance. The undergraduate assessments also show a collective relationship with the internship measurement, even when students’ pre-medical school differences were held constant. Such findings can help educators understand which measures best predict clinical performance.

Authors

• Ming Lee, University of California - Los Angeles

• Paul F. Wimmers, University of California - Los Angeles

• Michelle Lynn Vermillion, University of California - Los Angeles

• LuAnn Wilkerson, The University of Texas - Austin