Examining the Generalizability of an Automated Scoring Method and Identifying Student Difficulties With Scientific Explanations

• In Event: 45.027 - Supporting Elementary and Middle School Students to Write Scientific Explanations

Abstract

Purpose
The present study sought to investigate the generalizability of a method to automatically score students’ scientific explanations as well as explore student difficulties with explanations.

Theoretical Framework
The Next Generation Science Standards (NGSS, 2013) emphasize practices that students should master, including constructing explanations. Prior studies on written explanations involving a claim, evidence, and reasoning (CER) have demonstrated that students often experience a number of challenges generating all three types of statements (cf. McNeill et al., 2006). These studies, however, use rubrics that do not operationalize explanations at a fine-grained level, which is necessary in order to pinpoint the difficult structural components of each type (namely, C, E, and R). Nor is the typical level of scoring in others’ work fine-grained enough to give teachers precise formative data about their students’ explanations or do real time scaffolding of students’ explanations.

Recently, our group has developed automated scoring for CER scientific explanations based on a rubric that operationalizes the subcomponents for claim, evidence, and reasoning statements (see Appendix C; Authors, 2017a, 2017b), written at the end of a science inquiry investigation in Inq-ITS (Authors, 2013). Here we seek to investigate the generalizability of our automated scoring method, developed for Density, when applied to a new scientific domain, Lunar Phases.

Methods
Participants included 115 middle school students who constructed scientific explanations at the end of an Inq-ITS Lunar Phases activity. The automated scoring method from Authors (2017b) was modified for the Lunar Phases activity (see Appendix C) and applied to student explanations. Two human raters also scored explanations using the modified rubric (see Appendix C) and reached high agreement for all subcomponents of CER (kappa > .8). Final agreed-upon scores were used for analyses, (described below).

Data Sources
Scores generated by the Inq-ITS’ automated method were compared to human raters’ scores, and also examined to identify areas of student difficulty in the domain of Lunar
phases.

Results
Results indicated high agreement between humans and the automated method (kappa > .80) for all CER subcomponents (see Appendix C). Automated scoring confirmed similar student difficulties with CER subcomponents found for Physical Science (namely, Density; Authors, 2017b), namely, 55% of students did not report sufficient data for the independent or dependent variable in their evidence statements, and 71% of students did not indicate a scientific theory in their reasoning. Based on findings such as these and with access to this fine-grained, automated, scalable scoring method, teachers will be able to identify and address the specific structural components that are difficult for their students in generating claim, evidence, and reasoning statements. In the future, automated scaffolds could be designed and provided to students in real-time based on the automated evaluation of their explanations.

Significance
This study provides fine-grained data about the specific difficulties students face when generating scientific explanations, as well as how these difficulties generalize across topics. A major contribution of this work is its potential for auto-scoring students’ scientific explanations at a fine-grained level and at scale, as is our goal for Inq-ITS.

Authors

• Rachel Fallon Dickler, Rutgers University

• Haiying Li, Rutgers University

• Janice Gobert, Rutgers University