Generative AI for Feedback and Item Generation: Research Findings

• In Event: GenAI and Adaptive Tutoring: Insights From Case Studies and Research Findings

Abstract

Despite advances in reducing the time required to create an Intelligent Tutoring System (ITS), the process remains prohibitively expensive and quite labor-intensive. Developing high-quality content for an ITS often requires several stages of curation, refinement, and evaluation. Educational platforms like the ASSISTments and OATutor projects cite spending two to five hours training authors, after which authors take ten to thirty minutes authoring each problem.

With recent advancements in generative AI, we leverage the emerging affordances of LLMs for the automatic curation of math (Elementary Algebra, Intermediate Algebra, College Algebra, and Statistics) hints using ChatGPT. We find that ChatGPT-based hints (i.e., worked solutions) in math lead to statistically significant learning gains compared to a no-feedback control and show no statistically significant differences from human-generated hints (anonymous et al., 2024). We also discuss a hallucination mitigation technique for LLMs, named self-consistency (Wang et al., 2022), and its efficacy in reducing ChatGPT’s error rate. Our investigation shows that this technique greatly reduces the hallucination rate, achieving near 0% hallucinations for algebra topics (anonymous et al., 2024). We further apply the insights from our research to practical scenarios through direct deployments in STEM classroom settings. In this session, we additionally evaluate a recent research publication discovering that algebra questions generated by ChatGPT are on par with or better than gold-standard textbook items, with higher discriminating power (Bhandari, Liu, & Pardos, 2023). The overarching aim of the discussion is to illustrate how the emerging affordances of generative AI can be effectively harnessed to transform STEM education.

Authors

• Shreya Bhandari, University of California - Berkeley

• Zachary A. Pardos, University of California - Berkeley

• Ioannis Anastasopoulos, University of California - Berkeley