Tagging speech segments by adults and children from daylong recordings in preschool using Artificial Intelligence

• In Event: Poster Session 04
  In Poster Session: PS 04 Topic - Technology, Media & Child Development

Abstract

Introduction
Despite a growing body of evidence that children’s early language growth is shaped by the quantity and quality of the language that they hear, there is a lack practical tools to assess a child’s language environment. Thus, educators and parents must use their own anecdotal estimates to decide if an early language delay could be due to limited opportunities for interactions with adults and peers, and whether or not parent/educator intervention is leading to measurable improvements in a child’s language environment.
The current project is focused on developing Artificial Intelligence that uses speaker recognition algorithms to quickly quantify key indicators of a child’s language environment, such as utterances by a primary child, peers, and adults. Our dataset consists of spontaneous conversational speech recorded with LENA units (www.lenafoundation.org) worn by children in a high quality childcare learning center in the United States. The first step in this task would be to categorize ‘who spoke when’ and is known as diarization. However, we are focusing on ’speaker group’ diarization which includes classifying speech segments being from adults or children, from across multiple classrooms. The child wearing the LENA device on his person is referred to as Primary Child (PC) and rest of the children as Secondary Children (SC).

Dataset
A total of 79 hours of speech and non-speech child and adult data was transcribed by CRSS transcription team. We have selected ~32 ½ hours of speech audio for training, development and test sets for training diarization system and evaluating its performance. A total of 19,125 utterances (~14 hours) are in training set, 38,666 utterances (~6 hours) are in development set and 33,418 utterances (~12 ½ hours) are available for testing set. Thus, the data had 16 hours of adult audio while ~7 ½ hours of PC and ~8 ½ hours of SC.

Method
Deep Learning is a subfield of Artificial Intelligence that utilizes Deep Neural Networks (DNNs) to learn complex functions from data. We have used a DNN that specializes in learning bandpass frequency filters from the audio to classify it to belong to different speakers. The system block diagram (Fig. 1) presents our approach for learning the models using training data and evaluating them on testing set, based on best performance on development set. We utilize transcripts as oracle segmentation and the classification task is to map quick conversational turns from the LENA audio data as being from PC, SC or Adults (AD).

Results
Confusion matrices for the three speaker groups are presented (Fig. 2). Here the horizontal rows represent true labels, while the vertical columns represent the predicted labels of the utterance-level segments. Thus, the diagonal of this matrix contains the correct percentage of predictions by our model which are 80.4% for PC, 75.6% for SC and 79.1% for AD speaker group.

Conclusion
Thus despite the data imbalance in favor of AD, our SincNet model achieves better performance for PC, which displays its generalizability. This is one of the best performance (Kothalkar et al., 2019) for child speech diarization.

Authors

• Prasanna Vasant Kothalkar, University of Texas at Dallas
  Presenting Author

• Dwight Irvin, University of Kansas
  Non-Presenting Author

• Jay Buzhardt, University of Kansas
  Non-Presenting Author

• Beth Shanks Rous, University of Kentucky
  Non-Presenting Author

• John Hansen, University of Texas at Dallas
  Non-Presenting Author