Using Neural Networks for Automated Classification of Elementary Mathematics Instructional Activities

• In Event: Automated Classroom Observation With Machine Perception

Abstract

Objective. Recent advances in computer vision, machine learning, and deep learning may provide solutions to challenges involved in using video at scale in research on teaching quality and teacher preparation. In this study, we drew on 1,000 hours of videos of elementary mathematics instruction to investigate a number of ways that deep neural networks can be employed to classify instructional videos.
Perspective. A deep neural network is a hierarchical learning structure that tends to learn more complex and abstract features of a given set of data in its deeper layers. With this structure, a deep learning network can approximate complicated decision functions that directly map input data to output labels. Recent advances in deep learning indicate that it is possible to train computers to classify basic activities in videos of elementary mathematics instruction. For example, a computer could potentially determine whether a teacher is lecturing or facilitating classroom discussion. However, little is known about the feasibility of having a computer discern more complex instructional activities in a lesson such as whether teachers’ responses to students’ questions promote or constrain further student dialogue and engagement.
Data and Methods. Our study utilized videos of elementary instruction from the Responsive Classroom study and the Development of Ambitious Instruction study. In each study, about 150 elementary teachers were observed teaching mathematics four times each for about 45 minutes to an hour. Thus, the study presented here had access to 1,200 videos and 1,000 hours of video of elementary math instruction.
In this study, we developed 10 object labels and 20 activity labels for mathematics lessons. The activity labels were based on the Mathematics-Scan (M-Scan) classroom observation instrument; M-Scan includes nine features of instruction organized within four domains: mathematical tasks, discourse, representations, and coherence (Berry et al., 2013; Walkowiak, 2010; Walkowiak et al., 2014).
In 2019-20, members of our research team used these labels to view all 1,200 videos and assign object and activity labels to the edited video; 70% of the labeled videos were used to train the neural networks to classify objects and activities; and 30% of the labeled videos were used to assess the accuracy of each network in terms of classifying objects and activities.
Findings. We trained a neural network using the output of an object detection network, the output of an activity recognition network, and captions created with a dense captioning technique featuring a long short-term memory network (LSTM) structure. We found that this neural network was more accurate in assessing complex instructional activities (based on M-Scan) than networks that only used labels provided by object detection and activity recognition.
Significance. This study evaluated multiple types of neural networks for automated classification of mathematics instructional activities in classroom videos. Results shed light on the accuracy of neural networks and the possibility of using these methods on a large scale. Along with making significant gains in efficiency while significantly reducing costs, this work has the potential to yield new developments in teaching metrics and autonomous classroom simulations.

Authors

• Peter A. Youngs, University of Virginia

• Ginger S. Watson, University of Virginia

• Scott Acton, University of Virginia

• Samarth Singh, University of Virginia