Facial Features for Automatic Detection of Student Affect With Newton's Playground in the Wild

• In Event: 63.059 - Emerging Perspectives on Understanding Learning Behaviors in Digital Environments

Abstract

Objectives
It is widely acknowledged that affective states are ubiquitous to learning and can facilitate or hinder learning outcomes (D'Mello, 2013; Pekrun & Stephens, 2011). Real-time affect detection is needed to facilitate the study of affect and to develop learning environments that respond to affective states as they occur. The present objective was to advance computerized detection of affect in authentic educational contexts. We used automatically computed facial features (e.g., brow wrinkling, mouth movements) to detect engagement and frustration of students as they interacted with Newton’s Playground, an educational game that assesses and promotes learning of conceptual physics (Shute, Ventura, & Kim, 2013).

Theoretical Framework
Developing technologies to detect affect during learning is an interdisciplinary endeavor that spans psychology, the learning sciences, and computer science. The facial features we use are based on the Facial Action Coding System (FACS) (Ekman & Friesen, 1978). FACS describes facial muscle movements using a variety of Action Units (AUs) such as AU4 (Brow Lowerer) and AU26 (Jaw Drop). AUs have been shown to relate to affect in learning environments (McDaniel et al., 2007), a link that is grounded in embodied theories of affect (Keltner & Ekman, 2000).

Methods and Data Sources
Data were collected in a high school in the Southeast United States, where 132 students interacted with Newton’s Playground in their school’s computer lab for approximately 55 minutes per day for two days. Videos of the students’ faces were recorded with commercial webcams during the interaction. At the same time, affect was annotated by two trained human observers using the Baker-Rodrigo Observation Method Protocol (Ocumpaugh, Baker, & Rodrigo, 2012). The present study focused on engagement and frustration, as these two states comprised 90% of the affect observations. Affect detectors were developed by applying machine learning techniques that inferred the affect annotations from automatically computed facial features.

Results and Conclusions
The accuracy of the affect detectors was evaluated by comparing the computer’s predictions of affect to the human annotations. Each detector discriminated engagement or frustration from all other affective states. Detectors were trained on data from a random subset of students and tested on data from the remaining students to ensure detectors would generalize to new students. We were able to detect engagement with an accuracy of 68% (19% better than chance) and frustration with an accuracy of 76% (13% better than chance), thereby demonstrating the promise of our approach.

Significance
This research is significant because we have, for the first time, developed fully automated detectors of engagement and frustration during interactions with an educational game in the wild - a school computer lab. Our future work will include testing these detectors in other learning environments and with different student demographics in order to study their generalizability. Automatic detection of affect will allow researchers to scientifically study affective states as they arise without exclusive reliance on self-reports, and will allow computerized learning environments to adapt to the affective experience of students.

Authors

• Sidney K. D'Mello, University of Notre Dame

• Nigel Bosch

• Ryan Baker, Teachers College, Columbia University

• Jaclyn Ocumpaugh, Worcester Polytechnic Institute

• Valerie J. Shute, Florida State University

• Matthew Ventura, Florida State University

• Lubin Wang, Florida State University

• Weinan Zhao, Florida State University