Predicting Medical Exam Question Difficulty: Embedding, Machine Learning, and Feature Impact

• In Event: AI for Higher Education Assessment

Abstract

Background: Accurate item difficulty prediction is crucial for high-stakes medical examinations. However, prior research shows inconsistent findings regarding the most effective modeling components.
Objective: This study systematically investigates the impact of domain-specific embeddings, input content granularity, and choice of machine learning regressor on predictive performance.
Method: This research analyzed 2,815 medical multiple-choice questions, comparing various embedding models, feature combinations, and machine learning regressors (e.g., XGBoost).
Results: XGBoost outperformed other models. A domain-specific embedding model consistently improved accuracy. Using only the item stem and correct answer provided the optimal balance between predictive accuracy and model parsimony.
Conclusion: These findings offer significant, actionable insights for developing data-driven measurement practices in medical education.

Authors

• Shicong Feng, Peking University

• Tianpeng Zheng, Peking University

• Hao Hang, National Institute for Communicable Disease Control and Prevention of Chinese Center for Disease Con

• Jiayi Liu, Peking University

• Zhehan Jiang, Peking University