Role-Based Generative AI for Equitable Formative Feedback: Mixed-Methods Evidence from a Web-Based Course

• In Event: AER Roundtable Session Friday 1:45 pm - Gold 4
  In Roundtable Session: Human–AI Collaboration and Dialogic Learning

Abstract

Generative AI promises scalable formative feedback, yet equity and validity remain uncertain. We embedded a five-agent GPT-4o ensemble—evaluator, equity monitor, metacognitive coach, aggregator, reflexion reviewer—inside a web-based reflection platform for a 12-session AI-literacy course (28 adults, 84 reflections). Mixed-methods analysis combined trace analytics, Bayesian accuracy estimates, and coder think-alouds. The system matched human graders (MAE = 0.47; QWK = 0.46), halved the worst-band error gap (ΔMAE = 0.50) benefiting lower-performing learners, and produced feedback rated 3.97/5 for usefulness. Scoring took 7.7 s—11× faster than humans—at $0.0016 per reflection. Findings show role-based LLMs can deliver motivational, bias-aware feedback at scale, pointing to practical pathways for equitable learning analytics.

Authors

• Chenyu Zhang, Massachusetts Institute of Technology

• Xiaohang Luo, University of Pennsylvania