Evaluating the Validity and Reliability of General Surgery Milestones 2.0: A National Study

• In Event: Pushing Measurement Forward: Application of Advanced Quantitative Methods

Abstract

We evaluate the structural validity and reliability of ACGME Milestones 2.0 ratings within general surgery using 2020-2021 national data, which represent 7,650 residents from 328 programs. Using multilevel confirmatory factor analysis (MCFA), we tested a two-level, one-factor model based on parceled milestone domain ratings. MCFA revealed that all six core competencies loaded strongly onto a single latent construct of overall competence at both the within-program (resident) and between-program levels, stratified by postgraduate year and evaluation period. Generalizability theory provided additional evidence of rating reliability. These findings provide strong support for the internal structure validity and reliability of the Milestones framework, validating the use of item parcels to represent core domains in assessing general surgery resident competence.

Authors

• Dandan Chen, Mass General Hospital/Harvard Medical School

• Jonah Thomas, Massachusetts General Hospital