Toward Detection of Learner Distractions in a Medical Learning Environment: A Subgroup Discovery Approach

• In Event: 44.081 - Data-Driven Techniques That Complement Theoretical Study of Cognition and Learning With Technology

Abstract

Purpose and Theoretical Perspective. A key feature inherent to solving multifaceted ill-structured problems, such as diagnosing a clinical case, is the prospect of multiple paths to the correct diagnosis (Author, Date); resultantly, this opens up increased possibilities for distractors. Thus, to provide the requisite scaffolding in such cases, an adaptive feedback mechanism to address learner misconceptions is of particular relevance. Learning analytics (LA) has become an important part of such efforts (Berland, Baker, & Blikstein, 2014; Winne & Baker, 2013). In this vein, this paper takes a step in addressing this issue, by using a LA technique called subgroup discovery (SD; Wrobel, 1997) to unpack the rules for how learners link evidence items to potential diagnoses in BioWorld (Author, Date), a learning environment for developing clinical reasoning skills.
Methods and Analysis. Thirty volunteer undergraduate students participated. Participants solved three endocrinology cases (Amy, Cynthia, and Susan Taylor) in BioWorld. The analysis in this paper uses the log-data generated by BioWorld.
The SD task was set to generate rules that account for how learners link evidence items to potential diagnoses in BioWorld. In the first step, the algorithm generates rules that account for incorrect diagnoses made while submitting their solution for any of the three cases based on the linguistic features that characterize symptoms highlighted from the case description and linked to a hypothesis. In the second step, the algorithm generates rules that account for the fact that the linked hypothesis is in fact a distractor, meaning that the learner linked a particular symptom to an incorrect hypothesis during task performance (i.e., differential diagnosis).
Results. Despite having the results for all three cases, which we intend to provide in the session, we limit the interpretation of results to the Amy case (correct diagnosis: Diabetes Mellitus (type 1)) in this synopsis. Forty terms were extracted from the symptoms highlighted by the learners using a series of text pre-processing steps (case transformation; filtering for length of terms and stopwords; and, tagging for isolating parts-of-speech). Table 1 shows the rules obtained from both steps of the SD task. The terms “cleaning”, “room”, and “homework” resulted in the most precise predictions of incorrectness of the final case solution (i.e., Precision: 80%-83%). The step 2 results confirm that the diagnostic error was made throughout the differential diagnosis. A closer inspection suggests that Pregnancy, Crohn’s Disease, Food Poisoning, and Diabetes Mellitus (type 2) are listed in the differential diagnosis as these symptoms (i.e., “…homework and cleaning her room is an extreme effort) can be thought of as distracting the learners from the correct solution to the case.
Conclusion. The findings illustrate that the SD algorithm identifies learner misconceptions during a differential diagnosis and its impact towards task performance. These findings can be useful for developing mechanisms in BioWorld for detecting the identification of relevant symptoms that distract learners in order to deliver prompts (e.g., lab-test that was missed or a relevant set of symptoms that may have been overlooked by the learners) that are intended to address them.

Authors

• Tenzin Doleck, McGill University

• Eric G. Poitras, University of Utah

• Susanne P. Lajoie, McGill University