Enhancing Scenario-Based Assessment of Science Inquiry Through Automated Conversations

• In Event: 63.057 - Automated Assessments in Science Inquiry

Abstract

This work leverages existing work from the field of intelligent tutoring systems (i.e., AutoTutor; Graesser, Person, Harter, & Tutoring Research Group, 2001) to enable the exploration of automated conversations for assessment purposes (Jackson & Zapata-Rivera, 2016). More specifically, a scenario-based assessment which included simulations and two automated conversations with virtual characters (called the volcano scenario; Zapata-Rivera, Liu, Katz, Bauer, & Vezzu, 2013) was augmented with three additional automated natural language conversations. The original conversations asked students to compare notes and to (dis)agree with a prediction made by the virtual peer character. The three new conversations require students to explicitly discuss the causal relation between the amount of seismic data being collected and how that affects the quality (or accuracy) of the prediction of the likelihood of volcanic eruption. Asking students to express these causal relations in their own words may provide additional assessment information on how and what they choose to say. Thus, the current study focuses on the potential impact of incorporating the three new automated conversations.
Current Study
A study was conducted to explore some of the potential assessment benefits and drawbacks of incorporating automated conversations focusing on causal and evidence-based reasoning. A school-based study was conducted with 115 middle school students (6-8th grade) from three states (IL, AR, UT). Participants completed a pre-survey, interacted with a version of the volcano scenario (with or without the additional causal reasoning conversations), and then completed a post-survey. The pre-survey included questions on demographics, individual differences, and prior knowledge in science. The post-survey included questions similar to the pre-survey, but also included additional constructed response, multiple choice, and figure-labeling questions related to science knowledge and reasoning.
Results
Analyses were conducted to examine potential post-survey differences on science outcomes and user experience between students who did and did not engage with the additional causal reasoning conversations.
A one-way ANOVA was conducted on the proportion of correct responses from the selected response post-survey science questions focusing on content knowledge. Post-survey selected response scores were not significantly different between students who did (M = .54, SD = .22) and did not (M = .56, SD = .19) have the causal reasoning conversations, F(1,113) = .298, p = .586.
A one-way ANOVA was also conducted on the post-survey constructed response science items focusing on science reasoning. Students who engaged with the causal reasoning conversations (M = .93, SD = .56) did not score significantly different on the science reasoning questions from students who did not get the causal reasoning conversations (M = .88, SD = .49), F(1,113) = .261, p = .610.
Conclusions
Based on the initial results from the study, it does not appear that the additional causal reasoning conversations affected students’ content knowledge or science reasoning performance on the post-survey. These initial findings are promising from an assessment perspective because the new conversations may provide more information about students’ science inquiry and causal reasoning without inadvertently affecting student knowledge. This outcome warrants further investigation into the conversational evidence collected from the summary conversations.

Authors

• Tanner Jackson, Educational Testing Service

• Diego Zapata-Rivera, Educational Testing Service

• Blair Lehman, Educational Testing Service

• Lei Liu, Educational Testing Service