Assessment of Expert Decision Making in Medical Diagnosis

• In Event: Transfer for Learning in the Classroom: Developing Contexts to Examine STEM Learning Skills and Strategies

Abstract

When expert doctors diagnose patients, they employ “predictive frameworks,” or mental models of the relationships between disease symptoms and their possible causes. Doctors often refer to these frameworks as “illness scripts.” Medical students, however, are rarely explicitly trained or assessed on their use of such predictive frameworks. In this project, we interviewed expert doctors to identify the decisions they make when applying their predictive frameworks to diagnose a patient. We then developed a relatively straightforward medical diagnosis problem that mimics the real-life diagnosis process and requires students to make key decisions. Students are asked to identify key features of the illness presentation, then decide what additional information to collect (which aspects of physical exam to focus on, tests to order). They decide how to interpret the new information and explain what information leads them to their eventual diagnosis. The assessment requires them to choose potential diagnoses at four different points during the process, so we are able to see how quickly they narrow down to the correct diagnosis. The questions that ask them to explain their diagnoses (potential or final), and the questions asking them to explain their choice of tests, clearly reveal whether they are relying on well-developed predictive frameworks for their reasoning process.

We piloted the assessment with 73 medical students, spanning first to fourth year, and five expert physicians. Students at all levels demonstrated their lack of well-developed predictive frameworks. In explaining their diagnosis and decisions, they tended either to reference only minimal information or to recite excessive, irrelevant, details, instead of focusing in on key, relevant, information. Compared to experts, they also spent much less time planning what information they needed and more time reviewing results. As a consequence, their process in coming to a diagnosis was less efficient. Only around half of students reached the correct diagnosis; those who did needed more information (test results) than experts before they settled on the correct diagnosis. Between experts, there were also illuminating differences depending on whether the case was within their specialty (pediatrics). The three pediatricians put the correct diagnosis at the top of their possible diagnoses from the very beginning, and described how key pieces of information fit into their framework, which led to the diagnosis. The non-pediatricians (internists) didn’t settle on the correct diagnosis until much later, but it was on their list of possibilities from the beginning and their explanations were more deliberate than the students’. These experts, unlike students, demonstrated a clear predictive framework and consciousness of its limits, by explaining key features that they weren’t sure applied in the context of babies, which prevented them from ruling out certain alternative diagnoses earlier.

We are developing a new curriculum for a clinical reasoning course, in which second-year medical students practice making expert-like decisions and work through contrasting cases of disease presentations to develop explicit predictive frameworks. At the end of the course, we will test students with mystery cases in our assessment format to evaluate whether the curriculum improved their clinical reasoning process.

Authors

• Candice Kim, Stanford University

• Argenta Price, Stanford University

• Carl Wieman, Stanford University

• Sharon Chen, Stanford University