Exploring Key Features for Effective Machine Learning Scoring of Science Constructed Responses

• In Event: Measuring What Matters: Challenges and Opportunities in Assessing Critical Thinking

Abstract

With growing emphasis on scientific skills such as problem-solving, constructed response assessments are increasingly valued for revealing students’ knowledge-in-use. While machine learning (ML) offers promise for evaluating formative assessments, concerns about scoring reliability shaped by question-level measurement features, may hinder its broader use in science education. This study investigates how specific measurement variables influence ML model performance. Using data from the Next Generation Concept Inventory, which assesses student understanding of food, energy, and water systems through open-ended responses, we trained 168 text classification ML models across nine items and five measurement variables. Results indicate that questions focusing on water and question structure significantly affect model performance, as measured by Cohen’s Kappa.

Authors

• Heqiao Wang, Old Dominion University

• Kevin Haudek, Michigan State University

• Amanda Manzanares, Denver Museum of Nature & Science

• Caterina B. Azzarello, University of Northern Colorado

• Chelsie Romulo, University of Northern Colorado