Evaluating Next Generation Science Standard–Aligned Student Constructed Responses With Machine Learning

• In Event: Applying Machine Learning in Next-Generation Science Assessment

Abstract

Objectives
Crafting Engaging Science Environments (CESE), an international NSF funded study, implements Next Generations Science Standard (NGSS) aligned curriculum content, based on the principles of Project Based Learning (PBL), into classrooms across Michigan, California, and Finland. A study of this magnitude raises the issue of how to score assessments related to these holistic education standards, where memorization of facts is only a minor component of a student’s classroom experience (Authors, 2014). More comprehensive methods are necessary to gauge the ability of a student to engage in the creative, critical, thinking required to adapt to the rapidly changing demands of careers in the sciences and technology, which poses limitations for larger scale studies wishing to measure this deeper level of understanding (Haudek, et al., 2019). Through the use of machine learning, our study transcends the boundaries imposed by limited funding and human errors in scoring to gain deeper insights into students’ use of formative assessments.
Theoretical Framework
“Machine learning is one of the fastest growing areas of computer science with far reaching applications” (Mohri, Rostamizadeh, & Talwalkar 2018). This study applies machine learning to the assessment of an estimated 120,000 collected short answer responses. We employ AACR program that ensembles eight computer algorithms to accurately measure students’ understanding of their science curriculum.
Method and Data Sources
Approximately 7,500 students completed a summative test at the start of the school year containing several constructed response items before beginning the CESE curriculum, with more students responding to items designed to assess the CESE curriculum. Stata software was used to turn the responses into questions formatted for Qualtrics. Raters attended rigorous calibration meetings, where each rubric was introduced, and items were scored by consensus with the curriculum designer. Human raters were given these responses and scored each response into 3 bins representing beginning, developing, or proficient understanding. Scores were collected in small batches until high agreement in interrater reliability (IRR) was reached (Cohen’s Kappa >0.8). Once we obtained sufficiently high IRR, the consensus scores, and students' responses were given to the AACR team.
Results
Using feature extraction analysis, the AACR software examined these responses by their component features, or word combinations, and analyzed them using 8 different classification algorithms, each of which “voted” on the final score. AACR returned the initial results of a predictive model similar to that of human raters (Cohen’s Kappa 0.777) and continued to improve with progressive additions of new responses and human scores (Cohen’s Kappa 0.785).
Significance
The CESE curriculum shows the promise of sustained interest in the future study of STEM, and provides a unique opportunity to examine a developing research methodology. As NGSS standards are adopted and enacted in the classroom, their benefits cannot be studied without meaningful assessments and an effective method to evaluate those responses (Cheuk, et al. 2019). For large scale analysis the use of machine learning algorithms and online survey software is an effective method to reduce costs incurred by scoring formative adaptive student responses.

Authors

• Sarah Yvonne Maestrales, Michigan State University

• Quinton Baker, Michigan State University

• Israel Touitou, Michigan State University

• Joseph S. Krajcik, Michigan State University

• Barbara Schneider, Michigan State University