Does Transformer Deep Learning Yield More Accurate Sores on Written Explanations Than Traditional Machine Learning? (Poster 2)

• In Event: Artificial Intelligence–Augmented Assessment Tools to Support Instruction in STEM

Abstract

Objective
Integrating the three dimensions of scientific knowledge— science and engineering practices (SEPs), disciplinary core ideas (DCIs), and crosscutting concepts (CCCs) creates challenges for developing assessments as more sophisticated performance-based assessments are needed yet are time-consuming to score. One solution is using ML to score student responses (Zhai, 2021). This research uses a state-of-the-art approach and compares it with the traditional ML approach. We ask: Does transformer deep-learning using BERT yield more accurate scores on student written explanations than logistic regression or naive Bayes algorithms?
Perspectives
More than ten types of algorithms are broadly applied in science assessments (Zhai et al., 2020), which yielded machine-human agreements (MHAs) ranging from k = 0.09 to 0.97 (Zhai et al., 2021) This large deviation suggests a need for more sophisticated algorithms.
Bidirectional Encoder Representations from Transformers (BERT) is a deep-learning application that uses a multi-layer transformation to analyze and label text (Abadi et al., 2016). Unlike the previous ML, BERT has the advantage of needing less training data to achieve high MHAs. However, no evidence shows the advances of BERT over earlier algorithms in science education (Xue et al., 2021).
Methods and Data Sources
This study used three NGSS-aligned tasks developed to assess middle school students’ ability to use energy to explain science phenomena (NGSA, 2022). One-thousand students’ written explanations were rated using multi-faceted diagnostic rubrics to score assessment tasks that required the use of the three dimensions of scientific knowledge (example, see Appendix A). Trained content experts scored students’ responses using a rigorous procedure and achieved high inter-rater reliability: Cohen’s k and Fleiss k were above 0.70.
The data were partitioned into three files- training 60% of responses, validation 20% of responses, and testing 20% of responses and were used for algorithms development.
Results
The diagnostic rubrics yielded 23 score points for the three items; thus, we developed 23 scoring models using three algorithms (see Table 1). Repeated measures ANOVA suggests a statistically significant difference between the three algorithms’ scoring accuracy (F(2,44) = 42.742, p < .001). Post hoc tests using the Bonferroni adjustment demonstrated that accuracy for BERT is statistically significantly higher (M = 93.32, SD = 3.92) than for naive Bayes (M = 84.94, SD = 5.98) or logistic regression (M = 89.44, SD = 4.99), p < .001. Cohen’s k, Greenhouse-Geisser correction suggests a statistically significant difference between the three algorithms (F(2,44) = 12.438, p < .001). Post hoc tests using the Bonferroni adjustment showed that k for BERT is significantly higher (M = .731, SD = .266) than for naive Bayes (M = .595, SD = .183) or logistic regression (M = .646, SD = .226), p < .01.
----------------------------------------------Table 1---------------------------------------------------------
Scholarly Significance
This study demonstrates that BERT can yield statistically significant better results in MHAs for constructed responses in science education than traditional ML approaches. Additionally, the range of Cohen’s k was narrower than in other studies but still represents an area for growth when working with ML for automated scoring applications.

Authors

• Holly Kennedy Amerman, University of Georgia
  Presenting Author

• Xiaoming Zhai, University of Georgia
  Presenting Author

• Ehsan Latif, University of Georgia
  Presenting Author

• Peng He, Michigan State University
  Presenting Author

• Joseph S. Krajcik, Michigan State University
  Presenting Author