Automatic Objective Measurement of Student Emotions in Computer-Enabled Classrooms

• In Event: Emotions in Technology-Rich Learning Environments: Theory, Measurement, and the Future

Abstract

It is widely acknowledged that emotions are ubiquitous to learning and can facilitate or hinder learning outcomes. Consequently, the last decade has witnessed an explosion of research into the antecedents, behavioral correlates, and consequences of academic emotions (Pekrun & Linnenbrink-Garcia, 2014). Unfortunately, the scientific study of emotion has been stymied by a lack of objective measures, leading to a near exclusive reliance on self-reports. There has been advancements in physiological/behavioral emotion measurement, but much of this work has been restricted to labs. Our objective was to use facial expressions (e.g., brow wrinkling) and interaction patterns (e.g., click streams) to automatically measure confusion, frustration, engaged concentration, boredom, and delight while groups of students interacted with Physics Playground, an educational game for conceptual physics (Shute et al., 2013).

The theoretical grounding of the behavioral measurement approach lies in embodied theories of cognition and affect (Niedenthal, 2007). These theories posit a tight coupling between emotional states and bodily responses (physiology, facial expressions, and overt actions). The methodological foundation is in the field of affective computing (Calvo et al., 2015), where sensors and algorithms are developed to automatically monitor bodily signals, and machine learning methods are used to detect latent emotional states from these signals.

Data were collected in a Southeast United States high school, where 133 students interacted with Physics Playground in their school’s computer lab for approximately 55 minutes per day for two days. Videos of the students’ faces were recorded with commercial webcams, and log files recorded details of the unfolding interaction (e.g., screen content, student actions). At the same time, affect was annotated by two trained human observers using the Baker-Rodrigo Observation Method Protocol (Ocumpaugh et al., 2012). Facial features were computed from the videos using computer vision techniques. Log files were distilled in order to identify patterns of student behavior that might be diagnostic of their emotions. Supervised learning techniques were used to build emotion detectors that reproduced the observer reports from the facial- and interaction-features, both individually and in concert.


Accuracy was computed as the degree of alignment between the computer’s emotion predictions and the observers’ reports. Overall accuracy was modest when evaluated with the A′ metric (.67 or roughly 34% greater than chance). Face-based affect-detectors were more accurate than interaction-based detectors. However, the face-based detectors were only applicable to 65% of the cases where the face could be identified in the video. A combination of face and interaction features improved the applicability of detectors to 98% of cases without sacrificing accuracy. Follow-up analyses confirmed that the detectors generalized across students, gender, ethnicity, class period, and day of week.

Measurement is a precursor to change. This research is significant because we have, for the first time, developed fully automated emotion detectors for use in noisy computer-enabled classrooms. Automatic emotion measurement allows researchers to more systematically study academic emotions at a fine-grained level and at scale. It will also allow digital learning environments to adapt to student emotions in real-time, thereby affording enhanced levels of personalization than previously possible.

Authors

• Sidney K. D'Mello, University of Notre Dame

• Nigel Bosch

• Shimin Kai, Teachers College, Columbia University

• Luc Paquette, Teachers College, Columbia University

• Ryan Baker, Teachers College, Columbia University

• Jaclyn Ocumpaugh, Worcester Polytechnic Institute

• Valerie J. Shute, Florida State University