Using Machine Learning to Automate Analysis of Written Explanations of Intermolecular Forces

• In Event: Applying Machine Learning in Next-Generation Science Assessment

Abstract

Paper 5. Using Machine Learning to Automate Analysis of Written Explanations of Intermolecular Forces
Objectives
A core idea in chemistry is that atoms and molecules interact with one another through intermolecular forces (IMFs) that result in macroscopic phenomenon. In this study we develop and test a computational model to automate the analysis of student explanations of IMFs. Instructors can then use these resources to modify their course to better support 3D-learning in their classroom.
Theoretical Framework
Prior research has shown that students struggle to understand the causes and consequences of IMFs (Peterson, Treagust, & Garnett, 1989; Williams, Underwood, Klymkowsky, & Cooper, 2015). In our work, we have focused on helping students develop a deep understanding of interactions using the lens of three-dimensional science learning (which consists of disciplinary core ideas, crosscutting concepts, and scientific practices) (National Research Council, 2012) by engaging students with constructing models and explanations. By designing tasks that ask students to predict or explain we can gather powerful evidence about what students can do with their knowledge. However, analyzing such student writing is not always possible for instructors due to the large numbers of students involved: in these cases machine learning can provide instructors meaningful feedback on what their students can do.
Methods
In our earlier work we developed a task in which students explain how and why neutral atoms interact (Authors, 2016). The goal is to elicit an explanation of the formation of London dispersion forces (LDF), a type of IMF responsible for many observable phenomena (for example how geckos can walk up walls). We developed a scheme (Authors, 2016) to characterize the degree to which these explanations are causal mechanistic, a type of reasoning that connects phenomenon to the behavior of the entities a scalar level below (Krist, Schwarz, & Reiser, 2018). We coded undergraduate general chemistry student responses to develop computational models using machine learning algorithms from the Automated Assessment of Constructed Responses (AACR) web portal that is capable of analyzing large number of responses.
Results
To develop these models, we iteratively hand-coded sets of approximately 100 responses to train the AACR system until the human-computer agreement levelled off at a Cohen’s kappa value of 0.773 with 950 responses coded. With this well performing model we used the AACR system to automate the coding for nearly 8000 responses. To verify that we still had good human-computer agreement we coded a subset of the computer scored responses by hand. At this point, it appears that we have been able to develop and use a model to analyze previously uncoded explanations of LDFs like a human could, but in a much faster manner.
Significance
This marks an important step towards developing a robust tool that instructors could use to get meaningful feedback about how their students understand a difficult, but important idea like IMFs. As we move towards incorporating 3D-learning and integrating practices into our assessments, using machine learning in this manner can help us develop resources that instructors can use to further improve their teaching.

Authors

• Keenan Noyes, Michigan State University

• Melanie Cooper, Michigan State University