Improving Generated Feedback For Novice Teachers within DCS Through the Expertise of Teacher Educators

• In Event: Preparing 21st-Century Teachers With Digital Clinical Simulations: Advancing Equity and Innovation in Teacher Education

Abstract

The Teacher Moments platform enables teacher educators to provide targeted feedback to novice teachers during digital clinical simulations with generative pre-trained transformers (GPTs; Hillaire et al., 2022; Marvez et al., 2022). However, what is possible when that feedback is provided by a large language model (LLM) trained with the assistance of teacher educators’ pedagogical expertise? To explore this question, we developed a prototype that allows teacher educators without coding experience to train LLMs to provide feedback to novice teachers about a “moment” of classroom decision-making within Teacher Moments, such as responding to a student in a full group discussion scenario. We posit that by training LLMs with the perspectives of experienced educators who recognize the nuances associated with equitable teaching, the technology would become more educationally purposeful.

We used a scenario initially designed to promote decision-making aligned with ambitious and equitable mathematics teaching stances (Horn & Garner, 2022). First, the prototype showed teacher educators a sample of the decisions novice teachers made within a specific moment during the scenario, and prompted them to identify “characteristics” related to the type of in-the-moment feedback they would want to provide. For instance, in the context of the example above (responding to a student in a full group discussion), teacher educators may decide that feedback should contain characteristics such as explicitly validating the student’s thinking or asking them to share more ideas. For each characteristic a teacher educator defined as important, we used open-source GPTs to classify a set of pre-collected novice teachers’ responses as either having or not having that characteristic. For instance, if a novice teacher told a student they “are incorrect and need to try harder,” that response would ideally be classified as not containing the characteristic of validating the student’s thinking. As a result, that novice teacher would see a scripted piece of feedback in the simulation related to that classification. Teacher educators were asked to either approve how the GPT classified the novice teachers’ responses or edit the classifications accordingly. After an iterative calibration process, the teacher educators’ final classifications were used to fine-tune LLMs to provide nuanced feedback to novices during Teacher Moments simulations.

This paper will share the infrastructure design and preliminary results pertaining to how a group of teacher educators classified a set of novice teacher responses within a mathematics classroom scenario. Our goal for this work is to illuminate what happened when a group of teacher educators participated in this process. Broadly, this work informs how increasing transparency and perspectives can enhance the ability of GPTs to provide nuanced, in-the-moment feedback to teaching novices during fully-automated simulations.

Author

• Erin Barno, ETS