Predicting General Surgery Resident Competence: Evaluating Accuracy and Fairness of Statistical Learning Methods

• In Event: Pushing Measurement Forward: Application of Advanced Quantitative Methods

Abstract

This study aimed to develop and evaluate machine learning models to predict intraoperative competence among General Surgery trainees. We analyzed ratings from 3,793 trainees using data from the Society for Improving Medical Professional Learning OR. Seven classification models were trained and evaluated in terms of accuracy and fairness. The k-nearest neighbors (KNN) demonstrated the highest sensitivity (0.747) and F1 score (0.741) while Gradient Boosting Machines (GBM) achieved the highest accuracy (0.717). Support Vector Machines (SVM) had the highest specificity (0.744). Fairness analysis showed minimal gender disparities, with KNN showing the most balanced performance. Results were consistent in a subset of laparoscopic cholecystectomy procedures. These findings support the potential of machine learning models to enhance surgical education through data-driven performance assessment.

Authors

• Ting Sun, University of Utah

• Stella Yun Kim, University of North Carolina - Charlotte

• Gabrielle Moore, University of Utah

• Brigitte K. Smith, University of Utah