Evaluating Chatbot Performance on Board Continuous Certification Examinations: Methodological Considerations and Results

• In Event: Medical and Credentialing Test Applications

Abstract

The recent expansion of access to LLM-powered chatbots has raised questions about chatbot performance on certification and licensure assessments. This study evaluates the performance of multiple chatbot programs on a medical board continuous certification assessment and outlines the implications of various research design considerations when evaluating chatbot performance.

Authors

• Caroline Prendergast, AMERICAN BOARD OF SURGERY

• Andrew Jones, AMERICAN BOARD OF SURGERY

• Carol L Barry, American Board of Surgery

• Chandler Scott-Smith, American Board of Surgery