From Avoidant to Aware: Automating Feedback in Simulations on Equity in Computer Science Education

• In Event: A.I. and the Future of STEM Instruction: Designing New Models to Automate Feedback to Teachers

Abstract

Computer science (CS) enrollment has large imbalances in race, gender, and dis/ability among academic disciplines (Margolis, 2008). We designed a digital clinical simulation (DCS), Roster Justice (RJ), to help educators practice confronting inequities in CS education. In RJ, participants roleplay as a teacher discussing gender, racial, and dis/ability disparities in rosters for two math and CS classes with their principal (Borneman et al., 2020). Using previously collected data from an online course, we developed an artificial intelligence (AI) system to recognize when participants fail to identify equity issues and intervene with individualized feedback. We piloted this system in two education courses. Our aim was to determine whether participants found AI-automated feedback useful in their learning about equity.
RJ’s equity framework derives from Filback and Green’s adaptation of Milner’s opportunity-centered teaching (Filback & Green, 2013; Milner, 2020). This framework emphasizes how student demographics impact learning opportunities. RJ measures whether participants explicitly mention race/gender (“Aware”) or not (“Avoidant”) and allows participants to practice implementing equity-focused conversations in their schools.
We designed machine learning (ML) classifiers to determine the type of feedback participants receive. Previously collected text responses from RJ participants (N=541) were scored by raters as “Avoidant” or “Aware” (Borneman et al., 2020). We removed punctuation and trained multiple classifiers on the responses, finding that the Support Vector Machine Radial Basis Function model performed best (Bird, 2009; Hillaire et al., 2021; Pedregosa, 2011; Richardson, 2019). This model was very accurate with “Aware” responses (F-score of 0.91) and moderately accurate with “Avoidant” responses (F-score of 0.57; Table 1).
We then implemented the classifier within the RJ scenario to determine which type of feedback participants receive (Figure 1). We piloted the AI-integrated DCS with students in two education courses (N=17). Participants completed a survey about the usefulness of AI feedback. Simulation responses were scored by raters and compared to data from the online course without AI feedback (N=541).
We found initial evidence that AI-implemented feedback helped participants apply the “Aware” mindset. After receiving automated feedback in Prompt 1, pilot participants scored higher on all subsequent prompts compared to the original implementation without AI (Table 2). Additionally, three out of five participants who scored “Avoidant” on Prompt 1 revised their submissions after receiving AI-automated feedback, and the resubmissions scored higher in our equity framework. Also most participants (93%) rated the feedback as very or moderately useful on the survey.
This research examines AI feedback in large-scale learning environments. Participants perceived the suggested feedback as useful, scored better on the equity framework (Table 2) and revised their responses after receiving feedback. Our findings suggest that this feedback is successful at encouraging educators to enact a more “Aware” mindset. Thus, this AI integration can improve DCS learning of equity, aiding educators in creating more equitable learning environments. In future work, we aim to research other ways that ML can be used to offer personalized experience in practice-based teacher education.

Authors

• Aria Eppinger

• G.R. Marvez, Massachusetts Institute of Technology

• Joshua Littenberg-Tobias, Massachusetts Institute of Technology

• Garron Hillaire, Massachusetts Institute of Technology

• Sydney Dell

• Justin F. Reich, Massachusetts Institute of Technology