Artificial Intelligence Applications in Objective Structured Clinical Examinations

• In Event: Changing Trends in Assessments Across the Health Professions: Applications of Artificial Intelligence and Machine Learning

Abstract

Artificial Intelligence Application to Medical Education Assessment
Data features play a pivotal role in the success of the OSCE as an assessment method as examiners observe and assess various aspects of the medical students' performance at each station. These observations encompass communication skills, history-taking abilities, physical examination techniques, diagnostic reasoning, treatment planning, overall clinical decision-making, as well as data related to time management, ethical considerations, and patient-centered care. The richness of data collected during an OSCE allows for a comprehensive evaluation of the students' clinical skills and provides valuable feedback for their professional development. AI and Machine Learning hold great promise in revolutionizing the assessment process for undergraduate medical students during OSCEs, particularly in the automation of the scoring process. By training AI models on large datasets of previously scored OSCE encounters, these models can learn to evaluate and grade students' performances based on predefined criteria, similar to human examiners, streamlining the assessment process, reducing faculty burden, and potentially enhancing objectivity and consistency.
Current AI Application in Medical Education Assessment and Its Challenges
The current boom in applying artificial intelligence (AI) to medical education assessment has been going on for two decades.1 The existing literature primarily focuses on graduate medical education programs, especially in surgery, using automatic methods to evaluate technical skills.1,2 For instance, computer-vision-based models have been used to track surgeons’ hand and eye motions in simulated environments,3,4 and kinetics have been used to analyze muscle contraction.5,6 While a handful of studies extract features from physical examinations to assess medical undergraduates.7,8 others evaluate clinical reasoning skills through language models and written materials like electronic history records and patient notes.9–12 However, the lack of large annotated samples and difficulties in developing algorithms and detailed scoring rubrics can makes the modeling training process challenging.1,9–11
Another significant concern is the "black box" problem, where the decision-making process of AI models might be difficult to interpret or explain. In high-stakes evaluations like OSCEs, transparency, and interpretability are crucial to gain trust in the assessment outcomes. Additionally, it is essential to ensure that AI models are fair and unbiased, with training data being diverse, representative, and free from any inherent biases, to avoid unfair evaluations and adverse impacts on underrepresented groups, potentially perpetuating existing healthcare disparities. Moreover, the validation and continuous improvement of AI models in the medical education context require ongoing effort, regularly updating and evaluating AI systems to maintain accuracy and relevance. Rigorous testing against human assessments and ongoing monitoring for unintended consequences is essential to maintain the integrity of the assessment process.
Furthermore, the integration of AI in OSCEs requires significant infrastructure, resources, and expertise. Educational institutions must invest in technology, data management, and faculty training to effectively implement AI-based assessment methods. Overcoming these challenges will be crucial to harnessing the full potential of AI in assessing undergraduate medical students' clinical skills and ultimately improving the quality of medical education.

Author

• Chi Chang, Michigan State University