Using Generative AI in Education: Results from Large-Scale EdTech Platform R&D

Session Type: Coordinated Paper Session

Abstract

Generative Artificial Intelligence (GenAI) made a major technical breakthrough in 2022 with the public release of ChatGPT, DALL-E, LLAMA-2 and other applications that rely on large language models (LLM) to create new text/images in response to human prompts. This technology is early in the hype cycle, but is having substantial impacts in many industries, and education is one of the most promising areas identified for its application. Measurement experts and assessment service providers are familiar with LLMs and use them frequently in automated scoring and other contexts.

However, GenAI brings significant challenges to foundational principles in measurement. LLMs are inherently difficult to explain – even by the people creating them. Their training data is neither representative nor static, and in some cases is not even disclosed (e.g. GPT-4). In this context, we suggest that while the concepts of measurement are still important, their practice requires a change in methods. This session seeks to pierce the hype and hyperbole surrounding this new technology through papers that describe Item Development with Duolingo, Tutoring Dialogue with Khanmigo, Automated Distractor Generation & Item Feedback with EEDI, Math Feedback with NAEP Data, and interpretability research using eRater/Textevaluator.

This session is organized through NCME's Generative AI SIGIME.

Sub Unit

• Individual Papers, Innovation Demonstrations, Coordinated Sessions, or Organized Discussions

Session Organizer

• John Whitmer, Federation of American Scientists

Moderator

• Chris Ormerod, Cambium Assessment, Inc.

Individual Presentations

• Automated Item Development of Complex Reading and Listening Tasks for Language Assessments - Andrew Runge, Duolingo

• Automated Distractor and Feedback Generation for Math Multiple-choice Questions - Andrew Lan, University of Massachusetts at Amherst

• Providing dialogue-based stepwise feedback in mathematics - Kristen DiCerbo, Khan Academy

• Creating partially interpretable embeddings from large language models using best interpretable orthogonal transformations - Paul Deane, ETS

Discussant

• Magdalen Beiting-Parrish, Federation of American Scientists