Utilizing Mixed-Integer Programming Algorithms to Balance Examiner Severity Across Oral Examination Teams

• In Event: Educational Measurement Issues in the Health Care Professions

Abstract

Mixed-integer programming (MIP) algorithms provide a flexible optimization framework that can be applied to a variety of issues in testing. In testing, MIP algorithms have been used successfully to automate test assembly (ATA; Diao & van der Linden, 2011). With some modification, these algorithms can be used to balance examiner severity across multiple oral examination teams. Examiner teams were assembled for a medical certification oral examination utilizing a MIP algorithm aimed at balancing examiner severity. Results indicated improved reliability and less variability in severity across examiner teams compared to the previous year. Future research may examine whether MIP algorithms can improve the reliability and efficiency of other oral examination programs.

Authors

• Jason P. Kopp, Amazon

• Derek Sauder, American Board of Internal Medicine

• Beatriz Ibanez Moreno, American Board of Surgery

• Andrew Jones, American Board of Surgery