Automated Testing of Conversation-Based Assessments

• In Event: 49.057 - Conversation-Based Assessments

Abstract

Conversations with computer agents have been used to support student learning in the form of dialogue intelligent tutoring systems (Graesser, Person, & Harter, 2001; Chan & Baskin, 1990; Johnson, Rickel, & Lester, 2000). However, implementing dialogue systems to support these types of interactions is a complex process that requires several design iterations involving gathering data from users and using these data to refine the conversations. Although there are authoring tools that can support the development of these dialogue systems (Susarla, Adcock, Van Eck, Moreno & Graesser, 2003), testing of resulting conversations is usually a cumbersome process that can become a bottleneck in the development of conversation-based assessments. The main goal of this work is to speed up the development and the evaluation process of conversations-based assessments.
Borrowing on principles from software testing techniques (Davis, 1995), we have developed an approach for automated testing of conversation-based assessments. This process makes use of sample responses gathered from an interdisciplinary group of experts for predefined response categories that are represented in the form of a conversation diagram. These sample responses are used to create an xml testing script that serves as an input for a testing module that verifies that each sample response is correctly categorized. The testing module can evaluate both particular responses and complete sequences of responses (paths). Results of testing are used to refine conversation scripts. Once a testing script has been developed, it can be reused with many versions of the same conversation. It can also be used as a template for implementing new testing scripts for different conversations in various content areas.
The approach has already proven useful for identifying classification errors before trying the system with humans. This poster presents the results of applying this approach in the development of conversation-based assessments across different content domains (i.e., ELA, math and argumentation). Results include types of and amount of errors found, as well as estimated time savings compared to human testing.

Authors

• Irvin R. Katz, Educational Testing Service

• Diego Zapata-Rivera, Educational Testing Service

• Maria Bertling, Harvard University

• Tanner Jackson, Educational Testing Service

• Nathan Lederer, ETS

• Keith Kisser, ETS