Modeling Natural Facial Behavior With Computer Vision: Explorations in Learning Contexts

• In Event: Translating Scientific and Technological Advances From the Laboratory to the Classroom: It’s About Time

Abstract

Purpose: This talk describes novel translational research enabled by automated facial expression recognition technology.

Theoretical Framework:
Automated facial expression recognition systems enable new research into facial expression dynamics that was previously not feasible with manual coding methods, or which required application of electrodes to the face. Bartlett and colleagues have developed a state-of-the-art computer vision system called the Computer Expression Recognition Toolbox (CERT), which measures facial expressions in real-time. In addition to expressions of basic emotion, CERT measures the movement of 20 individual facial actions from Facial Action Coding System (FACS) (Ekman & Friesen, 1978) FACS is a system for objectively scoring facial expressions in terms of elemental movements, called action units, which roughly correspond with individual facial muscle movements. FACS was originally developed for manual coding by human experts. Manual coding is laborious, and can take up to 3 hours to manually code 1 minute of behavior. The frame-by-frame CERT output provides information on facial expression intensity and dynamics at temporal resolutions that were previously impractical with human coding. CERT has been successfully employed in a range of studies of spontaneous expressions (Bartlett & Whitehill 2010) and is available to other researchers for academic use (Littlewort et al., 2011). This talk will provide a demonstration of CERT, and then review two translational research projects currently underway at the Temporal Dynamics of Learning Center that employ this technology. The first project explores facial expressions of children during problem solving tasks (Littlewort et al 2011). The second project explores the development of adaptive tutoring systems that recognize facial expression and respond accordingly (Whitehill et al. 2011). Finally, a facial expression intervention for children with autism is introduced. This project will be described in the following talk by Tanaka.

Results:
In Study (1) Children’s facial behavior during problem solving organized into states with rapid transitions between them. We observed differences in facial behavior for correct versus error trials, both in terms of which facial muscles were moved, and in their dynamics. We can automatically predict from facial expression whether child will get the answer before they speak.
In Study (2), smiling was inversely related to the amount learned during a tutoring session. Those who smiled more learned less.

Scientific Significance:
Novel facial expression recognition technology is enabling new research paradigms that were infeasible previously using manual facial expression scoring methods. The two studies show that facial expressions of children during problem solving can be automatically measured with current technology. Both studies contribute to our understanding of facial behavior in problem solving, as well as make steps towards adaptive tutoring systems that recognize and respond to the state of the student. Computer vision technology is enabling new applications that respond to the student in real-time. The facial expression intervention for children with autism, described in the next talk, is an example of this new direction.

Authors

• Gwen Littlewort, University of California - San Diego

• Gwen Littlewort, University of California - San Diego