Influence of the COVID-19 Pandemic on Clinical Subject Exam Performance Over Time

• In Event: Loss or Gain? Evaluating the Impact of the COVID-19 Pandemic Across the Medical Education Continuum

Abstract

Purpose
Clinical clerkships are an integral part of medical education. After completing the pre-clerkship curriculum, medical students typically spend 4 to 8 weeks at a time gaining experience in different specialties. Most medical schools require clerkships in family medicine, internal medicine, neurology, obstetrics and gynecology, pediatrics, psychiatry, radiology, and surgery (Murphy, 2019). Clinical Subject Exams (NBME, 2021) are available in each of those specialties except for radiology, and a recent survey indicated that 95% of US medical schools use them for clerkships (AAMC, 2021). The purpose of this study was to analyze performance on clinical Subject Exams across medical schools over time to understand how the pandemic influenced student performance.

Perspectives
The COVID-19 pandemic necessitated widespread changes to the clerkship experience, from modifying student interaction with patients to introducing new components such as virtual simulations (e.g., Cain et al., 2020; Everard & Schiel, 2021). There were both disruptions and innovations. Using an interpretive grounded theory approach (Glaser, 2008), we will first consider changes in performance for each specialty, then draw conclusions about broader patterns.

Methods
The sample consisted of all students from US accredited medical schools who took one of the Subject Exams for end-of-clerkship purposes, with standard timing, between July 1, 2018, and June 30, 2021 (three academic years). Table 1 shows the analysis sample by specialty. The data were analyzed descriptively and through multilevel models. Initial models without predictors indicated that, across specialties, 7.6-10.0% of the variation in Subject Exam scores was due to differences between schools. The final model for each specialty included month, academic year, Step 1 score (grand-mean centered), and the interaction of month and academic year. All were treated as fixed predictors since the focus was on average effects across schools. Month and academic year were treated as categorical variables.

Results
Subject Exam results are equated, allowing longitudinal comparisons. Tables 2-8 list score averages and standard deviations (SDs) by month and academic year, and Figures 1-2 present the same information visually. Scores tended to increase as the academic year progressed. Additionally, there was a fluctuation in SDs, but no apparent pattern, such as consistently higher SDs in a specific academic year or quarter.
Table 9 shows a sample of the multilevel modeling results, including interpretations, based on family medicine. December was selected as the baseline month for comparisons as it is an approximate halfway point in the academic year (there is some variation in when the academic year starts for different schools). There were performance differences in relation to 2018-2019 in February through April in 2020 and March through June in 2021, suggesting the influence of the pandemic was periodic rather than sustained. For other specialties, like obstetrics and gynecology, there were no performance differences or, in the case of internal medicine, the differences consistently indicated score increases.

Significance
The insights from this study will help researchers and practitioners understand how the pandemic impacted student learning, how the impact varied from month to month in 2020 and 2021, and how it varied by specialty.

Authors

• Francis O'Donnell, NBME
  Presenting Author

• Carol A. Morrison, National Board of Medical Examiners
  Non-Presenting Author

• Irina Grabovsky, National Board of Medical Examiners
  Non-Presenting Author

• Annie Kochersberger, National Board of Medical Examiners
  Non-Presenting Author