Automated Measurement of Infant and Mother Duchenne Facial Expressions in the Face-to-Face/Still-Face

• In Event: Objective (Automated) Measurement of Infant and Child Behavior: Reliability, Validity, and Discovery

Abstract

The Face-to-Face/Still-Face (FFSF) assesses infant responses to parent unresponsiveness, an age-appropriate stressor. In the FFSF, parent and infant engage in face-to-face interaction (FF), the parent stops interacting with the infant but gazes at them with a neutral expression (SF), and the parent-infant interaction resumes (RE). Although still-face effects are well-studied, little is known about the degree to which the FFSF is associated with the production of intense affective displays. Duchenne smiles appear to express more intense positive affect than non-Duchenne smiles, while Duchenne cry-faces appear to express more intense negative affect than non-Duchenne cry-faces. The current study investigates whether proportions of Duchenne and non-Duchenne smiling and cry-faces differ across FF, SF, and RE episodes, using computer vision-based automated measurement.
Forty 4-month-olds and their mothers completed the FFSF protocol consisting of three 2-minute episodes: FF, SF, and RE. Infant and mother were separately video-recorded, and their recordings synchronized. Baby-FACS (certified) coders coded facial Action Units (AU4, AU6, AU12, AU20) in each video frame for the first 30 seconds of each FFSF episode. Automated facial affect recognition (AFAR) convolutional neural network (CNN) based computer vision software output the presence of the same facial AUs for the entire 2-minute episodes, indexing positive emotional expressions in infants and mothers, and negative emotional expressions in infants. Free-margin kappa between AFAR and FACS was above .63 for all mother and infant AUs over the first 30 seconds of the FF, the SF (infant only) and the RE for the entire sample, indicating acceptable reliability. Overall proportions of infants’ and mothers’ Duchenne smiling (AU6+AU12) and non-Duchenne smiling (AU12), and infants’ Duchenne cry-faces (AU4+AU6+AU20) and non-Duchenne cry-faces (AU4+AU20) were obtained.
With the first 30-seconds of each episode, indicated that expert coding and automated measurement of FACS AUs yielded similar patterns of change in infant and mother Duchenne and non-Duchenne expressions across the FFSF (see Figure 1). Full 2-minute AFAR analyses indicated that both Duchenne and non-Duchenne smiling declined from the FF to the SF (ps<.001), but only Duchenne smiling increased from the SF to the RE (p<.001; Figure 2). In similar fashion, both Duchenne and non-Duchenne smiling in mothers declined from the FF to the SF (ps<.01) and rose from the SF to the RE (ps<.001); however, the magnitude of mother Duchenne smiling changes over the FFSF were 2-4 times greater than non-Duchenne smiling changes. Infant Duchenne cry-faces increased from the FF to the SF (p=.03) and did not change between the SF and the RE (p=.27). Non-Duchenne cry-faces did not change significantly between the FF and the SF (p=.28) or between the SF and the RE (p=.19).
Expert coding and AFAR produced similar infant and mother Duchenne and non-Duchenne FFSF effects, highlighting the convergent validity of automated measurement. Substantively, this was the first description of Duchenne and non-Duchenne expressions in infant and mother during the FFSF. Duchenne expressions appear to be a sensitive index of infant and mother affective valence that are accessible to automated measurement and may be a target for future FFSF research.

Authors

• Yeojin Amy Ahn, University of Miami
  Presenting Author

• Onal Ertugrul Itir, University of Utrecht

• Sy-Miin Chow, The Pennsylvania State University

• Jeffrey Cohn, University of Pittsburgh

• Daniel Messinger, University of Miami