Automatic Coding of Parent-Infant Dyadic Play Videos Using Convolutional Neural Networks

• In Event: Poster Session 06
  In Poster Session: PS 06 Topic - Methods, History, Theory

Abstract

Head-mounted eye-trackers have successfully been used to assess parent-infant interactions in western, high resource settings. Such devices give us the flexibility to understand infant cognition in “the wild”, for instance, when parents name objects in their surroundings, word learning is facilitated by dyadic joint attention. However, researchers have typically relied on laborious manual coding in low resource settings, such as low- and middle-income countries (LMIC) due to a lack of technological infrastructure.

Here, we present a processing pipeline to assess parent-infant interactions in rural India and the UK, using low cost methodologies applicable to both high and low resource settings. Additionally, we employ state-of-the-art computer-vision methods that allow researchers to process large scale data collected from both high and low resource settings.

We demonstrate our method on data collected using Pupil Labs head-mounted eye-trackers in rural India and the UK, from 10-minute dyadic play sessions. Participants include 90 caregiver-infant dyads (six-month-olds) from the UK, as well as, High and Low SES dyads in rural India. The aim of the study is to create a pipeline that maps parents’ gaze position and infants visual experience in real world, where the field view is constantly changing, on objects, hands (own/other), and faces (other) using automated processing on both data sets.

At the core of our method lies deep learning, open source, convolutional neural networks (CNNs). Each network is specialised for a specific detection task, with separate networks for face & facial landmark detection (MTCNN), toy detection (FRCNN), and hand detection (MediaPipe). With the exception of toy detection with FRCNN our other networks require no further training. Using pre-trained networks suitable for our needs allows us to minimise training time, generally the largest computational cost for these projects.

We discuss how the outcomes can help the infant research community to expand their research in low resource settings and quantify parent-infant interaction across social contexts.

Authors

• Prerna Aneja, University of East Anglia
  Presenting Author

• Thomas Carr, University of East Anglia
  Non-Presenting Author

• Saber Sami, University of East Anglia
  Non-Presenting Author

• John P Spencer, University of East Anglia
  Non-Presenting Author