A Mashup of AutoTutor and Emotion-Sensing Technologies

• In Event: When Systems Collide: Challenges and Opportunities in Learning Technology Mash-Ups

Abstract

The Technologies. AutoTutor-ES (Emotion Sensitive) is a mashup of AutoTutor and affect sensing technologies (D’Mello & Graesser, 2012). AutoTutor is an automated computer tutor that helps students learn by holding a conversation with the student in natural language (Graesser, Jeon, & Dufty, 2008). AutoTutor’s dialogues are organized around difficult questions and problems on science or technology (e.g., physics, computer literacy, research methods) that require reasoning and explanations in the answers. The dialogue to answer a challenging question typically requires 50-100 conversational turns between AutoTutor and the student as they attempt to cover 3-7 key ideas. AutoTutor’s dialogue moves include feedback (positive, neutral negative), pumps for more information (“Tell me more”), hints, prompts to fill in missing words, summaries, corrections of student misconceptions, and answers to student questions. The system tracks the knowledge of the students and dynamically generates discourse to try to get the student to contribute more information and learn. AutoTutor was developed in the Institute for Intelligent Systems at the University of Memphis in an interdisciplinary team that included expertise psychology, computer science, education, computational linguistics, physics, and art. Dozens of experiments showed that AutoTutor helped learning compared to reading a textbook, with effect sizes that averaged 0.80 (Graesser et al., 2004; VanLehn et al., 2007).

Roz Picard at MIT had developed a number of noninvasive sensing technologies that tracked various emotions and affect states (Picard, 1997). The technologies successfully tracked facial expressions, body posture, mouse movements, and other overt behavior in an attempt to detect boredom, engagement, frustration, surprise, and many other emotions and intentional states.

The vision. The Memphis and MIT teams collaborated to build the emotion-sensitive AutoTutor-ES (D’Mello & Graesser, 2010, 2012; D’Mello, Picard, & Graesser, 2007). This was an important step because students experience a variety of emotions during complex learning, notably frustration, confusion, flow/engagement, boredom, surprise, delight, and anxiety (D’Mello, Craig, & Graesser, 2009). Three derivative discoveries and technologies emerged from the mashup. First, we discovered what emotions are prominent in complex learning activities, including how they interact with learning events, their timing, and their sequencing. Second, we developed technologies to automatically detect emotions on the basis of facial expressions, body movements, discourse, and speech. Third, we develop mechanisms that determined how the automated tutor would respond with appropriate emotions and discourse moves. These novel discoveries and technologies filled a glaring gap in education, psychology, and advanced learning technologies.

The Future. We are currently investigating the impact of AutoTutor-ES on the learning gains, motivation, and attitudes of learners. Different personalities of AutoTutor-ES have been developed, such as a polite empathetic conversational agent versus a rude confronting tutor. We are exploring how the student characteristics (personalities, prior knowledge) are good or bad matches with agents that have particular conversational styles. The hope is that complex learning will improve with tutorial dialogues that are sensitive to cognitive, affective, and social states.

Authors

• Arthur C. Graesser, The University of Memphis

• Sidney K. D'Mello, The University of Memphis