Comparing Trialogue-Based Tasks With Other Assessment Tasks

• In Event: 49.057 - Conversation-Based Assessments

Abstract

Conversational trialogues have been successfully integrated in intelligent tutoring systems (ITS) to motivate students in learning and assessment systems. A trialogue typically consists of two computer agents (a teacher agent and a student agent) and a human student. The current study compared an open-ended trialogue task with a parallel multiple-choice conversation and a multiple choice question followed by an open explanation (base case) in terms of student motivation and student performance. Although, multiple-choice questions have excellent psychometric properties and are used in many standardized test, many students have grown tired of these types of tasks. Evidence shows that by the time students reach high school they have increasingly negative perceptions of standardized tests, and ultimately decreased motivation to give genuine effort when completing tests [Paris et al., 1991].
The present paper tested two hypotheses: (1) open-ended trialogue tasks motivate students more and yield better task performance than multiple choice conversations and the base case providing better evidence of their knowledge/skills, and (2) the multiple choice conversational trialogue may motivate students and yield better performance than the base case.
It is probably that open-ended questions allow students to communicate with agents with more freedom in natural language conversations. However, it is likely that students feel more motivated in the multiple choice condition because it is easy to complete the task just by selecting the correct choice.
To test these hypotheses, we used the AutoTutor Script Authoring Tool for Assessment (ASATA) to develop a trialogue-based assessment system in the area of science inquiry. This system has game-like elements such as narrative and simulations. Tasks consisted of reviewing content knowledge about volcanoes, collecting and analyzing seismic data, making predictions about volcano eruptions, and evaluating the quality of the predictions. The only difference among the conditions was the type of conversation used (i.e., open-ended trialogue, multiple choice conversations and multiple choice questions followed by a short explanation).
150 college students will participate in this study with $25 payment, 50 in each condition. After login, they will accomplish a series of surveys, including demographic, test anxiety, prior experience with earth science, computer and video game usage, and a pre-test. Then, they will interact with the assessment system which involves self-assessment questions on engagement, motivation, and task difficulty. Finally, the student will complete a usability survey and a posttest.
Data will be analyzed with mixed modeling to compare the students’ task performance and motivation in the three conditions. Moreover, the variation in motivation, engagement and task difficulty will be tracked at different periods during the task. This variation may help us identify the more/less motivating activities, which ultimately help develop the more engaging assessment systems. Data collection will take place this fall, and results and discussions will be reported at the conference.

Authors

• Haiying Li, The University of Memphis

• Diego Zapata-Rivera, Educational Testing Service

• Blair Lehman, The University of Memphis

• Tanner Jackson, Educational Testing Service