Exploring Scalability Issues of Conversation-Based Assessments

• In Event: 49.057 - Conversation-Based Assessments

Abstract

New views of learning (e.g., constructivism and activity theory) call for innovative assessment modes that track students’ process of knowledge production. Conversation-based assessment such as trialogue is one promising assessment approach to track and record student higher-order thinking processes such as scientific inquiry, argumentation skills, and mathematical thinking. Through dialogical conversations, students engage in practices of professional mathematicians and scientists who share and develop ideas with colleagues through explanation and justification. Building off previous research on natural language intelligent tutoring systems (Graesser, Person, & Harter, 2001), trialogue approach entails computer-mediated conversations between the student and two virtual agents. Students may take different paths of scripted conversations based on the content of their responses. This project mainly focused on the scalability aspect of using trialogue as an assessment approach. We define scaling up as adapting an innovation successful in a certain context to effective usage in a wide range of contexts. We acknowledge that there is not a standardized method to explore the scalability of an educational innovation. Our research focuses on exploring the feasibility of producing multiple trialogue-based assessments in different domains and relevant cost and efficiency issues.
In this poster, to address scalability issues related to efficiency and generalizability, we describe how we scale up the trialogue assessment approach by: (1) building a parallel task to our first trialogue task – the Volcano task – in the science domain, and (2) adapting the trialogue approach in other domains such as mathematics and English Language Learning. To address the cost and efficiency aspect of scalability, we investigated the amount of reusable work across all trialogue projects, including storyboards, assets, programming codes, scoring method, conversation diagrams, rules, and regular expressions. We also documented the amount of time needed for developing various tasks including the staff time needed to conceptualize the task, to identify similar data sources and structures, and to create parallel versions. Additionally, for the two science tasks, we compared the degree to which the two tasks were parallel (e.g., identifying in storyboards where we were able to capture the same construct and where we were not), a number of additional ideas for parallel versions, and have begun to conceptualize a task model or common underlying conception that can be used to make connections between tasks.
To address the generalizability aspect of scalability, we designed an experimental study to investigate the generalizability capability of the trialogue approach by comparing the two science tasks (Volcano vs. Weather) to explore: (1) if students elicit similar levels of understanding in both tasks; (2) if there are learning effects after completing one of the two tasks. Single facet g analysis will be conducted and bivariate distribution of total scores and paths will be obtained. The results of the study shed light on major issues and provide implications for next steps to scale up the trialogue approach.

Authors

• Lei Liu, Educational Testing Service

• Isaac I. Bejar, ETS

• Diego Zapata-Rivera, Educational Testing Service

• Tanner Jackson, Educational Testing Service