Toward Automated Classroom Observation: Predicting Positive and Negative Climate

• In Event: Automated Classroom Observation With Machine Perception

Abstract

Objective. Classroom observation is a standard practice used to support educational research and teachers' professional development (Hamre et al., 2013; Mashburn et al., 2010). Its widespread deployment, though, is limited by the large cost of manually coding either live or video-recorded classroom sessions. This paper reports the results of machine learning experiments that were designed to explore architectures and training procedures in order to develop an automated classroom analysis instrument. More specifically, we developed a prototype of machine perception software that can analyze classroom observation videos from pre-kindergarten (toddler) classrooms and automatically estimate the Positive Climate and Negative Climate dimensions of the Classroom Assessment Scoring System (CLASS; Pianta et al., 2008).
Perspective. Machine learning is rapidly growing as a powerful technique in many fields that can identify subtle, non-linear interactions between different variables. In education, it has been employed to identify early warning detectors of student access (Kizilcec & Halawa, 2015), detect and track individual students’ emotions during class instruction and educational games (Baker et al., 2012), support curriculum recommendation systems (Pardos et al., 2017), and determine key moments during classroom instruction (Qiao & Beling, 2011).
Data and Methods. The CLASS is used characterize interactions between students and teachers independently of the particular subject matter being taught. To estimate CLASS dimensions using machine learning, we followed a multi-modal approach, where we analyzed both visual features of students' and teachers' facial expressions, as well as low-level audio features of the entire video, in separate time series models and in combined models. One overarching computational challenge during the project was how to harness modern "deep" perceptual architectures (e.g., based on convolutional and recurrent neural networks) despite the relative scarcity of labeled data compared to traditional deep learning application domains. We trained and assessed our neural networks using a subject-independent cross-validation procedure on a CLASS-coded dataset containing 241 videos of toddler classrooms collected at the University of Virginia.
Findings. The average accuracy of our system on videos not used for training was 0.40 (Pearson correlation) and 0.51 for Positive Climate and Negative Climate, respectively, and these numbers will likely improve given a larger set of training data. The automated system sometimes makes mistakes that a human coder would rarely make, and this is an important challenge to address in future research. Ablation analyses revealed that the auditory and visual channels of the videos contained complementary information for predicting CLASS scores.
Significance. This work represents the first automated system designed to detect specific dimensions of the CLASS. While our results were based on toddler classrooms, we are currently exploring the generalizability of the proposed machine learning architecture to elementary school settings. The longer-term goals of this research are to use an automated classroom observation system to provide teachers with more specific, accurate, and frequent feedback, and to develop more precise scientific instruments to support educational research.

Authors

• Anand Ramakrishnan, Worcester Polytechnic Institute

• Erin R Ottmar, Worcester Polytechnic Institute

• Jennifer LoCasale-Crouch, University of Virginia

• Jacob Whitehill, Worcester Polytechnic Institute