Text Mining in Written Scientific Argumentation Using Latent Dirichlet Allocation

• In Event: Modeling, Mining, and Visualizing Student Learning of Science and Engineering Practices From Digital Traces

Abstract

Argumentation (NRC, 2012) is a language-based science practice (Walton, Reed, & Macagno, 2008) that takes place in rhetorical writing situations. As such, it could be beneficial to explore the power of text mining to identify patterns in students’ argumentation. Text mining may also provide a means for making new discoveries to improve existing theories on student argumentation. In this study, we applied a novel text mining method—Latent Dirichlet Allocation (LDA)—to students’ written scientific arguments and used an interactive visualization technique to investigate two questions: (1) how does LDA detect underlying topics in students’ written scientific arguments?, and (2) In what ways do students’ explanations of data in support of their claim differ from their rationale for uncertainty involved in their argument?
LDA belongs to a category of latent variable modeling methods which identify topics from documents using a “bag-of-words” approach. In essence, LDA assumes that (1) a probability distribution over words in a document can be extracted as a mixture of topics, and (2) each topic itself is a probability distribution over a range of words. In our context, we treat each argument students write as a document. To transform each argument into a bag-of-words, we carried out several pre-processing procedures, e.g., removing non-letter symbols, common stop words, and stemming. To explore the results, we created interactive visualizations to represent underlying topics expressed in their arguments.
From the scientific argumentation task shown in Figure 4 (Lee et al., 2014), we collected 2,472 open-ended responses to the explanation prompt and 2,472 open-ended responses to the uncertainty rationale prompt. LDA successfully identified six topics in explanations (Figure 5) and five topics in uncertainty rationales (Figure 6). Our further analysis on a bag of relevant words belonging to each topic indicates that each topic in explanations uniquely represents a different way to coordinate evidence with theory, e.g. data only, reasoning only, data and reasoning combined, wrong reasoning types, and restatement of claim. Similarly, each topic in uncertainty rationales represents a unique way to cite uncertainty sources such as personal sources related to experience and confidence, and scientific sources related to correct vs. incorrect reasoning and data. We also observed a strong association between explanation topics and claim, χ2(5) = 97.6, p < .001. Similarly, five topics in uncertainty rationales have significantly different mean weights across students’ uncertainty rating scores, F = 19.74, p < .001.
The use of LDA greatly facilitates the processing of text data in a short time with almost no additional human coding efforts. LDA is completely automated, and its results have high replicability without involvement of the human bias and inconsistency associated with scoring. In contrast to other text mining methods such as automated scoring, LDA can be used to explore the patterns of student argumentation from the ground-up and investigate how students reason with data. This diagnostic potential can be further harnessed and leveraged to design feedback or actionable intelligence for teachers and students to make decisions on teaching and learning.

Authors

• Wanli Xing, University of Missouri - Columbia

• Hee-Sun Lee, The Concord Consortium

• Amy Pallant, Concord Consortium