Detecting WH keywords spoken by preschool teachers during science activities using Automatic Speech Recognition technologies

• In Event: Poster Session 13
  In Poster Session: PS 13 Topic - Technology, Media & Child Development

Abstract

Introduction
The quality and number of interactions children have with adults is a key factor in children’s language development in early childhood (Hart et al., 1995). The preschool classroom is a viable avenue for monitoring young children’s interactions with teachers and classmates. In such settings, teachers prompt exploration by asking questions that engage the children’s curiosity and experimentation, particularly in science-focused activities. Previously, the quality of teacher questions in the preschool classroom were captured in live or offline video observations (Sanders et al., 2016). This study represents one of the first efforts (Buzhardt et al., 2020) to monitor WH keywords (what, where, when, how) spoken by teachers in the classroom using Automatic Speech Recognition (ASR) technology.
Specifically, we present results of Keyword Spotting (KWS) algorithms applied to audio recordings from preschool settings for recognizing WH keywords spoken by adults in the preschool classroom. These recordings consist of unscripted speech recorded with LENA units (lenafoundation.org) worn by children in a high quality childcare learning center in the United States. The center had 7 teachers across 2 preschool classrooms.

Dataset
A total of 79 hours of speech and non-speech naturalistic data recorded by LENA units was transcribed by the CRSS-UTDallas transcription team. The ASR model is trained using a standard, phonetically-balanced, adult speech dataset. For testing the KWS model, 599 utterances (~20 minutes) of adult speech were extracted from daylong recordings based on the transcript information.
Method
The KWS algorithm in flowchart (Fig. 1) presents our approach for learning the models using training data and evaluating them on the preschool audio recordings. Typically, the KWS task involves recognizing if the keyword is present in continuous speech (Barakat et al., 2011) or the individual spoken word (Chen et al., 2014). This is performed by comparing ASR model scores for speech utterances and the target keyword.
In our case of continuous speech, a fixed length window of these scores for given keyword are moved over the length of each utterance in fixed steps using Dynamic Time Warping algorithm. If the distance at any step is below a predetermined threshold, it is recognized as the presence of the keyword in the utterance. For each utterance, this matching is performed with 5 randomly selected template audio files for each WH keyword.

Results
599 random adult utterances across all preschool activities are selected for evaluation of our Keyword Spotting system. Out of these, 348 utterances were recognized to contain WH keywords. Accuracy of WH keyword recognition in these 348 utterances is presented in Table 1. With a focus on analyzing teacher talk in science activity areas, accuracy of detecting WH keywords in 54 utterances in the area are also reported in Table 1.

Conclusion
Using speech recognition technology to identify WH keywords in preschool classrooms by frequency, content and location can help teachers better and more efficiently understand their interactions with children, make changes when needed, and monitor the impact of those changes. Better performance can be achieved by improving the ASR model or KWS technique.

Authors

• Prasanna Vasant Kothalkar, University of Texas at Dallas
  Presenting Author

• Dwight Irvin, University of Kansas
  Non-Presenting Author

• Jay Buzhardt, University of Kansas
  Non-Presenting Author

• Beth Shanks Rous, University of Kentucky
  Non-Presenting Author

• John Hansen, University of Texas at Dallas
  Non-Presenting Author