Automated Measurement of Affective Facial Expressions in 4- and 8-Month-Olds: Reliability with Manual Coding

• In Event: Objective (Automated) Measurement of Infant and Child Behavior: Reliability, Validity, and Discovery

Abstract

Facial expressions are among the earliest behaviors infants use to express their emotional states, and are crucial to preverbal social interaction (Darwin, 1877). Manual coding of facial expressions is a labor-intensive process involving human judgment, which may limit the analysis scope and reproducibility of results. Recent developments in computer vision have substantially advanced automated facial expression recognition in large adult samples, yielding rich and reproducible results at lower time-investment. Baby FaceReader 9 (Noldus, 2022) is a commercially available tool for automated classification of facial expressions but has not been validated with infant data. We compared Baby FaceReader 9 to manual coding of affective facial expressions using a longitudinal dataset. The facial expressions of 51 infants at 4 months (148 videos) and 53 infants at 8 months (141 videos) were manually classified as either positive, negative, neutral or missing (Cohen’s kappa of .85 and .91, respectively) during naturalistic face-to-face interactions with mother, father, and an unfamiliar adult.
Manually coded data were 29% positive, 66% neutral, 5% negative, <1% missing. Manual coding yielded a higher percentage of available data than the automated measurement: 99% compared to 85%. Results indicated reasonable classification accuracy (AUC = .81) but a poor correct classification rate (F1 score = .67) using Baby FaceReader 9’s global emotional valence formula to discriminate positive from negative/neutral manually coded facial expressions (see Figure 1). Chance-level discrimination (AUC = .49) and poor correct classification rate (F1 score = .69) were observed for discriminating manually coded negative from neutral facial expressions.
Bayesian logistic regressions were then used to assess whether a priori configurations of automatically detected facial action units (Table 1) could discriminate 1) positive from the pooled set of negative and neutral manually coded facial expressions, and 2) negative from neutral manually coded facial expressions. Action units indexing smiles (AU12), eye constriction (AU6), and mouth opening (AU25+AU26+AU27) were more likely to be automatically detected during positive than neutral/negative manually coded facial expressions. Notably, a one standard-deviation increase in automatically detected smile intensity was associated with a 100-fold increase in the odds of a manually coded facial expression being positive rather than negative/neutral. At higher smile intensities, eye constriction decreased the odds of the manual code being positive by 3%, whereas mouth opening increased the odds by 42%. Automatically detected cry-faces (AU20), brow lowering (AU3+4), eye constriction (AU6), and mouth opening (AU25+AU26+AU27) independently increased the odds of a negative rather than neutral manually coded facial expression between 40- and over 100-fold. These results provide initial support for applying Baby FaceReader 9 to automatically detect smile configurations and cry-faces as an index of positive and negative affect in young infants using FACS Action Units rather than the FaceReader 9 global emotional valence formula. The assessment of classification accuracy requires further replication in datasets where all three facial expression states have more balanced distributions; suitable scenarios are for instance those invoking more frequent and intense negative affect displays such as face-to-face/Still-face interactions or reactions to painful stimuli.

Authors

• Martina Zaharieva, University of Amsterdam
  Presenting Author

• Eliala Alice Salvadori, University of Amsterdam

• Daniel Messinger, University of Miami

• Ingmar Visser

• Cristina Colonnesi, University of Amsterdam