Evaluation of a Novel Simulation-Based Assessment of Clinical Reasoning Processes Using Multivariate G-theory

• In Event: Defining Competence: Milestones and Skills Assessment

Abstract

We evaluated the psychometric properties of scores derived from a novel multi-dimensional simulation assessment of three hypothesis-driven information gathering (HDIG) criteria: Agility, Characterization of the Chief Concern, and Curiosity. We used generalizability theory to evaluate HDIG score reliabilities and used generalized mixed modeling to evaluate predictive utility of HDIG scores on diagnostic accuracy. We found a) HDIG score reliabilities (individual and composite) were impacted substantially by number of cases than raters, b) composite HDIG scores with theoretical weighting were substantially more reliable than individual HDIG scores, and c) composite HDIG scores significantly related to diagnostic accuracy. These findings provide a foundation for using HDIG-based assessments to offer detailed feedback and enhance clinical reasoning training in medical education.

Authors

• Hyeryung Lee, University of Iowa

• Thai Quang Ong, National Board of Medical Examiners

• Su G. Somay, NBME