Paper Summary

Toward a Hidden Markov Modeling Framework for Real-time Assessment in Tutorial Dialogue

Sun, April 15, 12:25 to 1:55pm, Pan Pacific, Floor: Lobby Level, Oceanview 1&2

Abstract

A rich history of research on one-on-one tutoring has identified key components of tutorial dialogue strategies that support learning through tutoring. These strategies include types of adaptive cognitive scaffolding, motivational support, and collaborative dialogue patterns (Core, Moore, & Zinn, 2003; K. Forbes-Riley, Litman, Huettner, & Ward, 2005; Katz, Allbritton, & Connelly, 2003; Rosé, Bhembe, Siler, Srivastava, & VanLehn, 2003). All of these mechanisms are informed by real-time assessment, a hallmark characteristic of tutorial dialogue. This assessment includes tutors continually assessing students’ knowledge and skills in order to adapt effectively. In task-oriented tutorial dialogue, in which all tutorial guidance is provided in the context of a student solving problems while the tutor observes, real-time assessment plays a particularly critical role in ensuring that the student progresses through problem-solving episodes. Modeling and understanding real-time assessment techniques is an important step toward identifying optimally effective tutoring approaches.
Research on tutorial dialogue explores the cognitive and motivational strategies used by human tutors in a variety of domains. These lines of investigation have revealed regularities in the structure of natural language tutorial dialogues and have suggested hypotheses for why one-on-one tutoring is so effective. A promising research direction along these lines is to use corpora, collected from tutorial dialogue, to create computational models that can evaluate the differential effectiveness of tutoring strategies at a fine-grained level. There is growing evidence that meaningful tutorial dialogue patterns can be automatically extracted from corpora of human tutoring using machine learning techniques. A particularly promising family of computational models that offer significant potential for capturing the richness of tutorial dialogue is hidden Markov models (HMMs).
HMMs (Rabiner, 1989) are a promising modeling framework because they explicitly model the “hidden” structure of the rich interplay between tutors and students. HMMs operate in a “bottom-up” fashion, in contrast to the traditional approach of analyzing tutorial dialogue corpora from the “top down.” In a top-down approach, researchers begin with a set of tutoring strategies, usually informed by the literature or theory, and examine a corpus for patterns that fit the set of strategies. In contrast, in a bottom-up approach, theories from tutoring and natural language dialogue are used to inform corpus annotation at a low level, and then statistical models are built that induce the strategies inherent in the structure of the sequenced observations. HMMs are well suited to bottom-up analyses of tutorial dialogue.
The authors are engaged in an NSF-funded project on learning Java programming through tutoring that investigates HMMs for tutorial dialogue assessment. In this paper we provide an overview of the tutorial dialogue studies that have been conducted to investigate tutoring phenomena, describe the hidden Markov modeling approach that has been used to study these phenomena using machine learning techniques, and consider how hidden Markov models can be used to investigate real-time assessment.

Authors