Assessing Regulatory Patterns in Diagnostic Reasoning: A Precursor to Adaptive Scaffolding

• In Event: 63.033 - Scaffolding Metacognition

Abstract

Objectives: We examine the domain-specific nature of self-regulated learning (SRL; Alexander, Dinsmore, Parkinson, & Winters, 2011; Poitras & Lajoie, 2013; Schraw, 2007) as it pertains to diagnostic reasoning leading to patient diagnosis. In accordance with Zimmerman’s (2000) model, SRL is conceptualized as a cyclical process involving forethought, performance, and self-reflection, wherein feedback from prior performance leads to recalibration of subsequent efforts to solve a problem (Zimmerman & Campillo, 2003). This study examines ineffective patterns of SRL in medical students while they solve patient cases using BioWorld, a computer based learning environment designed to support diagnostic reasoning (Lajoie, 2009).

Methods: Patterns in diagnostic errors are discovered using subgroup discovery, a data mining technique (Herrera, Carmona del Jesus, Gonzalez, & del Jesus, 2011) that extracts patterns in relation to lab tests ordered by the novices and the accuracy of their diagnoses. The logs (11874 entries) corresponding to 30 users each solving 3 cases of varying difficulty, were aggregated into 304 lines of diagnostic reasoning delimited by instances where a diagnosis was selected. The data mining technique includes: a) discovery of patterns in the diagnostic tests ordered and errors committed; b) validation of patterns through metrics of diagnostic performance; and c) interpretation of patterns using a visualization of the diagnostic process. The subgroup discovery algorithm was set to extract rules that correctly included 70% of relevant examples as well as 5% of the total number of examples in the dataset.

Results: A total of four subgroups of errors in diagnostic reasoning were found, where non-pertinent lab tests were ordered while attempting to solve the most difficult case in BioWorld. The classification rules extracted to detect these errors covered 6% to 11% of lines of diagnostic reasoning with an average precision and accuracy of 76.6% and 44.4%, respectively. Novices that ordered these non-pertinent tests performed less well than others in terms of both efficacy and efficiency, showing lesser amounts of matches with the expert solution and greater amounts of lab tests ordered to solve the case (M = 3.2 and M = 14.9, respectively). A further examination of their impact on diagnostic reasoning suggests that these tests are linked with common confounding concepts in the diagnosis of Pheochromocytoma. Tests that are neither categorized nor prioritized in the later stages of problem-solving suggest that diagnostic errors are transient.

Significance: These findings have implications for scaffolding novices to regulate their own learning by monitoring the diagnostic process for common mistakes. Scaffolding prompts can remedy these issues by redirecting them to relevant topics within the library or delivering hints through the consult tool. Adaptive scaffolding based on this empirical work can lead to improved self-reflection while diagnosing patients with the goal of reducing error rates attributed to confirmation bias and overconfidence.

Authors

• Eric G. Poitras, University of Utah

• Susanne P. Lajoie, McGill University

• Tenzin Doleck, McGill University

• Amanda Jarrell, McGill University