AI-based Formative Assessments in Medical Education: Insight and Challenges

• In Event: Artificial Intelligence in Action: Transforming Assessments

Abstract

Developing high-quality assessments is time-consuming and resource-intensive, requiring deep understanding in content, cognitive complexity, and assessment purposes. In medical education, this is further complicated by limited clinical faculty time in team-based learning and rapidly evolving curriculum changes. Over two academic years (AY24–25 and ongoing AY25–26), weekly assignments—averaging 10 multiple-choice questions and one essay question—were delivered to 183 medical students. This study outlines item generation processes, quality control strategies, performance data, and evaluation results. Findings from AY24–25 suggest AI-generated questions hold significant promise, demonstrating content and format validity while offering cost and efficiency benefits. Future work should aim to enhance AI algorithms to better capture the complexity of clinical image generation, anatomical knowledge representation, and clinical reasoning.

Author

• Xiaomei Song, Case Western Reserve University