Visual Language Model-driven Recognition of Kindergarten Children Emotional Expressions During Competitive Games

• In Event: Empowering Young Learners: Enhancing Child Agency Through Educator-Facilitated Digital Exploration

Abstract

Objectives: Preschool teachers play a crucial role in teaching children how to be emotionally competent (Huang, Yang & Li, 2019). Current studies focusing on classroom students’ facial emotion recognition generally employed traditional machine learning or deep learning methods, not only are far from real world applicable; but also, mainly built on existing facial expression datasets (Negrão et al., 2021; Yu, 2021; Sharma & Mansotra, 2019). To address these issues, this research builds a children emotional expression recognition model by tuning a Visual Language Model (VLM) on a customized children expression video dataset obtained in competitive games (Alayrac, et al., 2022).
Method: This study used pre-training prompt learning to adapt a large VLM to the task of children's emotional expression. The video dataset of this study was collected in a demonstration kindergarten in Shanghai, with a total of 106 children aged 5-6 participating. To obtain typical, multiple, static, and dynamic emotional expressions of children, we designed four different competitive games, each presenting different winning and losing alternation modes. Children were randomly assigned to these four different groups. The video was recorded simultaneously fromfront and panoramic perspective.. Eachvideo was about 5 minutes longIn the construction of the emotion expression recognition model, we added a learnable prefix sequence, prompt word of the language model, describing children's emotions in competitive games to the front end of the input video for pre-trained VLM model. Then we used the gradient descent method to optimize this learnable sequence during the training of the children expression recognition model.
Results: A video dataset for training deep learning based recognition model of kindergarten children’s emotional expressions was constructed. We leverage the capacity of pretrained large VLM model by designing a learnable prompt training method in accordance with the characteristics of kindergarten children’s emotional expressions. The obtained children emotional expressions recognition model can be used in various real-world applications such as education robots. A three dimensional, i.e. time, arousal and valence, emotion expression and regulation strategies record for every child was obtained.
Implications: The primary importance of this study lies in optimizing a large-scale emotion recognition model specifically designed for young children, including those with special needs. The model can sense, capture, and identify young children's emotional expressions with high accuracy, recognizing both individual and combined emotions (Garner, et al., 2019). This capability allows for the development of a real-time, time-series model of each child's emotional expressions. Integrating this technology into kindergarten routines can help track and predict emotional patterns in children. Second, the model can detect and alert teachers to early signs of emotional outbursts in children with special needs, enabling timely and appropriate interventions. Third, the use of VLM model facilitates three-dimensional visualization of emotional expressions, encompassing valence, arousal level, and time. This represents a significant advancement in understanding young children's emotional expressions.

Authors

• Jin Liu, Shanghai Maritime University

• Huihua He, Shanghai Normal University

• Wenwei Luo, Shanghai Normal University

• Ilene R. Berson, University of South Florida

• Michael J. Berson, University of South Florida

• Mingji zhou, Shanghai Maritime University

• Hui Li, The Education University of Hong Kong