Using Automated Scoring to Promote Knowledge Integration in Science

• In Event: 66.030 - The State of the Art in Automated Scoring of Science Inquiry Tasks

Abstract

The Continuous Learning and Automated Scoring in Science (CLASS) project is investigating ways to guide students to integrate their ideas in science inquiry projects. Guiding inquiry learning requires building on the multiple, often conflicting ideas students bring to science class. Successful inquiry instruction can prepare students to autonomously make sense of new information throughout their lives. The CLASS project uses the Knowledge Integration (KI) framework to create rubrics, and the Web-based Inquiry Science Environment (WISE) to test automated scoring and KI guidance for four types of generative activities: short essays, drawings, concept diagrams, and graphs. Essay scoring uses c-rater—an analytic tool that uses an associative model based on a large database of previously scored responses. Drawings, concept diagrams, and graphs are scored by analyzing log files.
In this research, we scored all four item types using the KI rubric and assigned guidance based on the score. The rubric rewards linking ideas and evidence into a coherent argument and consists of five levels: the Off-task level includes irrelevant responses that are unrelated to the question being asked, such as “I don’t know;” the No-Link level includes non-normative ideas; Partial-Link Level indicates that students stated isolated relevant and normative ideas that are not meaningfully connected to each other; the Full-Link level is assigned to a response that includes one scientifically valid link between ideas and evidence; the Complex-Link level is assigned to a response that includes two or more scientifically valid connections between ideas and evidence.
CLASS developed a framework for designing personalized guidance that uses the KI score for each item. KI guidance acknowledges progress such as for the Partial Link level, “you have some ideas,” suggests next steps, such as for the Partial Link level, to make links by revisiting an activity or conducting a new activity. In four classroom experiments with seven teachers and 897 students, CLASS compared guidance designed to promote KI to three alternative types of guidance typically given in a middle school classroom. Drawing on data from pretests, embedded assessments, computer log files, posttests, and interviews, CLASS found that the KI guidance was more effective than general motivational guidance and specific guidance, and as effective as guidance designed by experienced teachers. KI guidance helped students use evidence from earlier in the unit to sort out ideas, and freed teachers to support students who needed extra help.
Automated scoring technologies enable teachers to enact inquiry activities in large classes while still providing the guidance necessary to ensure that students benefit from the instruction. Of course, creating successful guidance results from extensive trial and refinement with large numbers of diverse students. And, scoring algorithms require tuning to achieve acceptable levels of accuracy. In addition, these systems are not yet capable of scoring essays for high stakes tests. Nevertheless, these technologies have great potential for improving educational outcomes in science and other disciplines.

Authors

• Marcia Linn, University of California - Berkeley

• Libby Gerard, Mills College

• Ou Lydia Liu, ETS

• Kihyun "Kelly" Ryoo, University of North Carolina - Chapel Hill

• Anna Rafferty, University of California - Berkeley

• Jon Vitale, University of California - Berkeley