Automatic Monitoring of Emotions During Learning: From the Research Lab to the School Computer Lab

• In Event: Interdisciplinary Approaches for Analyzing Data From Multiple Affective Channels With Computer-Based Learning Environments

Abstract

It is widely acknowledged that emotions are ubiquitous to learning and can facilitate or hinder learning outcomes (D'Mello, in press; Schultz & Pekrun, 2007). Real-time emotion detection is needed to scientifically study emotions during learning as well as to develop advanced learning environments that can respond to emotions as they occur. The objective of this talk is to synthesize our research on fully automated systems to detect learner emotions. We will focus on systems that provide fine-grained emotion monitoring in controlled lab settings as well as systems that are scalable to computer laboratories in classrooms.

Developing technologies to detect emotions during learning is an interdisciplinary endeavor that spans psychology, the learning sciences, and computer science. Our research incorporates perspectives that view emotions as expressions, embodiments, outcomes of cognitive-appraisal, social-constructs, and products of neural circuitry (Calvo & D’Mello, 2010; Izard, 2010). These theories specify how emotions arise from appraisals of internal and external events, how emotions are subjectively experienced, and how they are expressed via physiological, bodily, and behavioral changes.

Data sets were collected in two studies where 55 learners completed 32-45 minute tutorial sessions with AutoTutor, an intelligent tutoring system with conversational dialogues (Graesser, Chipman, Haynes, & Olney, 2005). Videos of the learners’ faces and computer screens along with various sensor data (see next section) were recorded during the learning session. Approximately 6,500 instances of boredom, flow/engagement, confusion, frustration, delight, surprise, curiosity, and neutral were annotated by humans from the videos. Emotion detectors were developed by applying machine learning techniques that inferred the emotion labels from the sensor data.

The accuracy of the emotion detectors was evaluated by comparing the computer’s predictions of emotions to the human-annotated labels. Systems that monitored facial expressions, body posture, and contextual cues were correct approximately half the time and generalized to new learners (D'Mello & Graesser, 2010). Systems that relied on peripheral physiological measures (heart rate, skin conductance, muscle movements) yielded similar accuracies, but did not generalize to new learners (AlZoubi, D'Mello, & Calvo, 2012).

These systems’ reliance on sophisticated sensors makes them limited for use in computer labs in schools. To address this, we have started developing systems that use soft sensors, such as simple webcams (embedded in most laptops) to monitor facial expressions, body movements, and distance from the screen, and context models that are inferred from interaction dynamics (keystrokes, mouse movement, timing, and system feedback). Initial results with models that couple bottom-up sensor data with top-down contextual information are quite promising.

This research is significant because we have developed fully automated emotion detectors for fine-grained emotion monitoring in research lab environments. We have also made considerable progress towards developing scalable systems that might be used in computer labs in schools. These systems will allow researchers to scientifically study emotions as they arise without exclusive reliance on self-reports as well as to develop next-generation learning technologies that are adaptive to learner emotions.

Authors

• Sidney K. D'Mello, University of Notre Dame

• Arthur C. Graesser, The University of Memphis