Validating ECFMG Certification Policy Changes Using ACGME Milestones

• In Event: Advancing Competency-Based Assessment: Evidence-Based Decisions

Abstract

The purpose of this study was to explore whether internationally trained physicians (IMGs) certified through the current Educational Commission for Foreign Medical Graduates (ECFMG) certification process are comparable to those certified through the previous certification process, as measured by outcome of their first Accreditation Council for Graduate Medical Education (ACGME) Milestones ratings in residency.
Ensuring the competence of medical students entering residency has historically been accomplished through educational and examination requirements. ECFMG certification is required for all IMGs to apply for graduate medical education (GME) in the United States. One of the certification requirements is that IMGs should demonstrate adequate clinical and communication skills, a role previously filled by the United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS) exam. In response to the suspension of the Step 2 CS exam in 2021, ECFMG changed the clinical skills requirement for ECFMG certification (22, 23). Determining whether the current ECFMG certification process is as effective as the previous certification process in screening IMG applicants to U.S. GME training programs is critical (24).
In this retrospective study, we analyzed the six-month ACGME Milestones ratings during the intern year (PGY1) for 13,621 IMG residents who began the Internal Medicine residency training in an ACGME-accredited program between 2021 and 2023. ACGME Milestones ratings were selected as GME programs use the ACGME Milestones ratings to track and assess the development of trainees’ clinical competencies throughout the training (25–27).
We examined the Patient Care (PC) and Interpersonal and Communication Skills (ICS) competencies. These competencies were directly assessed in both the current and previous certification (Step 2 CS) requirements. The outcome measures are the six-month Milestone ratings (as deviation from the program mean) during PGY1. The focal explanatory variable is the ECFMG certification process IMGs went through to get ECFMG certified: the previous (took Step 2 CS) process vs. the current process.
Descriptive statistics were used to report the mean and standard deviation of the deviation ratings, stratified by trainees’ characteristics. Generalized Estimation Equation was utilized to examine the comparability between the current and previous certification requirements based on the milestone performance, accounting for the effect of trainees’ characteristics.
The study was deemed exempt by the Institutional Review Board at the University of Pennsylvania (protocol #853975).
ECFMG certification process (previous vs. current) wasn’t a statistically significant predictor of PC or ICS core competency. Failing the Step 2 CS exam was a statistically significant predictor of the Milestones deviation ratings for the competencies examined, where IMG residents who failed the Step 2 CS exam before scored slightly lower in the deviation rating on the PC core competency (β = -0.10, p = 0.004) and the ICS core competency (β = -0.08, p = 0.028).
The findings indicated that IMGs’ clinical competencies assessed through the ECFMG certification processes were not impacted by the change in certification requirements. IMGs screened by the current process received comparable Milestones ratings to those screened by the previous process, providing validity evidence to support certification decisions.

Authors

• Shiyao Yuan, FAIMER, a Division of Intealth

• Kenji Yamazaki, Accreditation Council for Graduate Medical Education