Conversation-Based Assessment of Science Inquiry Skills: Designing and Scoring Issues

• In Event: 66.030 - The State of the Art in Automated Scoring of Science Inquiry Tasks

Abstract

Drawn from the sociocultural and social constructivist perspectives on cognition, scientific discourse is a critical tool for science learning (Duschl & Osborne, 2002). If enculturation into scientific discourse is significant to scientific knowledge construction and externalization, it becomes imperative to use conversation-based assessments to collect evidence about how students are developing their knowledge and skills through communicating. In our work, by applying the principles of Evidence-Centered Design (ECD; Mislevy, Steinberg, & Almond, 2003) and leveraging advances made through previous research on natural language intelligent tutoring systems (e.g, Auto Tutor; Graesser, Person, & Harter, 2001), we developed an interactive scenario-based task – the Volcano scenario – that includes conversations among humans and virtual agents. The scenario and embedded conversations implement realistic situations (e.g., collaborative project-based research) to assess students’ scientific inquiry skills.
At the beginning of the Volcano scenario, students receive instruction and are introduced to the relevant seismic events that precede volcanic eruptions. Then they travel on a virtual field trip to collect data and engage in several conversation-based tasks. These conversations were designed to gather information on students’ data collection and data-based reasoning skills (e.g., how many seismometers did they place). Specifically, students interact with two virtual agents (Dr. Garcia and a student agent Art) to review and compare one of the student’s own notes with one of Art’s notes. Art’s note is created based on the student’s notes and used to gather additional evidence of data collection skills (i.e., understanding that more data provides better evidence). These conversations include 2 or 3 cycles of the student receiving limited feedback and having a chance to elaborate on their reasons for making a particular decision.
In total, there are four constructs being measured and scored throughout the volcano scenario. The constructs being addressed involve data collection, making data-based predictions, evidence-based reasoning, and communication. Within the conversation-based assessment, different paths of conversations were scripted based on expert judgment and designed to address particular aspects of the target constructs. Students are directed down particular paths based on the content of their responses (e.g., complete, partial, incorrect, irrelevant, metacognitive, metacommunicative; Zapata-Rivera, 2013; Zapata-Rivera, Jackson, Liu, Bertling, Vezzu, & Katz, 2014). The system utilizes the path information to implement an automated partial credit scoring rubric. For example, students who answer the main question correctly without any scaffolding are assigned full credit for the relevant construct(s), whereas students who do require scaffolding are assigned partial credit depending upon the amount and type of assistance required. Scores for each pertinent construct are assigned at the closing statement for each path. Initial partial credit scores are refined based on additional information coming from other sources, for instance reviewing available tools in the interactive environment.
The current study explores the validity of path-based automated scoring by comparing the automated scores with human ratings. In the full paper, we report on a study conducted through Mechanical Turk with 500 users. Human raters will code a subset of this data and we will compare their scores with the automatic path-based scores.

Authors

• Lei Liu, Educational Testing Service

• Tanner Jackson, Educational Testing Service

• Diego Zapata-Rivera, Educational Testing Service

• Jiangang Hao, ETS

• Isaac I. Bejar, ETS