Deep Learning Approaches to Classifying Teacher Questions Within A.I.-Based Classroom Simulation (ACTS): A Validation Study

• In Event: Automated Classroom Observation With Machine Perception

Abstract

Objective. This study assessed the use of deep-learning, active learning, and weak supervision to facilitate machine-assisted labeling and analysis of teacher questions within ambitious mathematics instruction towards the development of an artificial intelligence (AI)-based classroom simulation (ACTS).
Perspective. Next-generation classroom teacher preparation simulations require machine understanding for the generation of robust, content-specific classroom dialogue. A major obstacle to developing such simulations is the requirement for large training datasets of specific tasks. Active learning and weak supervision approaches have the potential to draw on less specific data to train neural networks. However, research is needed to assess the efficacy of these approaches with content-specific classroom dialogue and to examine the extent to which these approaches support the development of intelligent classroom simulations.
Method. To train ACTS neural networks, three qualified raters classified teacher questions drawn from mathematics classroom transcripts using an adapted version of the Instructional Quality Assessment (IQA; Boston & Candela, 2018) as (a) probing/exploring mathematical meaning /relationships, (b) procedural/factual, (c) other mathematical, or (d) non-mathematical. Because human coding of audio transcripts is labor-intensive, we tested transfer learning approaches in deep learning (Devlin et al., 2019), which requires less data for text classification (Kim, 2014), by initially labeling a small amount of data and then gathering validity evidence via an active learning approach (Settles, 2009) to automatically label additional data.
We then used active learning to classify teacher questions. Data labeled by raters were used to train the model along with a variety of additional sources including teaching videos, recorded chats, and YouTube transcripts. We assessed the effectiveness of these deep learning approaches as a substitute for manually labeling teacher questions using inter-rater reliability with one to two raters along with a classification label suggested by the active-learning, relying on an additional rater in cases of disagreement in classification. This was continued until a recommended classification was reached.
Once acceptable validity was established, the active learning model was implemented with new data and assessed for its accuracy to correctly classify teacher questions. In situations where the model had low confidence, rater scores were used to further train the model with continuous feedback in select scenarios.
Findings. Initial training of the neural network with limited data (n=250) resulted in moderate levels of accuracy of the human coding (K=.48) used to train the network to classify IQA categories for teacher questions. It took multiple iterations with additional data to reach K≥.7 for categories associated with ambitious mathematics instruction (e.g., probing/exploring). Analysis of the deep learning steps indicated reasonable model accuracy and identified the benefits of more training data and refined interpretation of the IQA categories.
Significance. This study examined the efficacy of a deep neural network to classify teacher questions necessary to support ambitious mathematics instruction in an AI-based classroom simulation. The findings support the broader use of iterative deep-learning, active learning, and weak supervision to facilitate machine-assisted labeling of questions in such instruction. This approach has the potential to support broad, scalable teaching practice and performance feedback in ACTS.

Authors

• Jim Bywater, James Madison University

• Debajyoti Datta, University of Virginia

• Maria Phillips, University of Virginia

• Ginger S. Watson, University of Virginia

• Sarah Lilly, University of Virginia

• Don Brown, University of Virginia

• Jennifer L. Chiu, University of Virginia