Innovations and Practices of AI and Language Models in Large-Scale Assessments

Session Type: Coordinated Paper Session

Abstract

Although not new, artificial intelligence (AI) gained extreme popularity with the introduction of ChatGPT in Fall of 2022. It resulted in great excitement and anxiety, particularly for educators. Advances in AI, natural language processing (NLP), and machine learning are now central topics in educational assessment. A proliferation of research and practices have explored the use of these techniques throughout the assessment process. While the potential for automatic item generation (AIG) and automated scoring were promising, concerns arose about privacy and the exclusion of human roles and professional expertise. Many fear the use of automated techniques and decision-making will result in a lack of interpretability of assessment scores and black boxes of accountability. Challenges have also arisen in ensuring the psychometric properties and the fairness of AI-based scores. The focus of this coordinated session is to investigate the advances of AI and NLP in large-scale assessment settings from research to practice. Over four papers, we explore several issues in evaluating the item quality within AIG, the use of language models in flagging learners in danger, and the use of AI in providing assessment feedback in practice. Attendees will improve their understanding of AI as it becomes increasingly prevalent in education.

Sub Unit

• Individual Papers, Innovation Demonstrations, Coordinated Sessions, or Organized Discussions

Session Organizer

• Shenghai Dai, WASHINGTON STATE UNIVERSITY

Individual Presentations

• Using Large Language Models for Evaluating Item Quality in Large-Scale Assessments - Guher Gorgun, University of Alberta; Okan Bulut, University of Alberta

• The Importance of Emotion Recognition for Identifying Learners in Danger in Large-scale Assessments - Joshua Southerland, Pearson; Luis Alejandro Andrade, Pearson VUE; Lee Becker, Pearson; Kyle Habermehl, Pearson

• Providing Standards-Based AI-Generated Feedback to Students on Their Writing - Susan Lottridge, Cambium Assessment, Inc; Sherri Woolf, Consultant; Chris Ormerod, Cambium Assessment, Inc.; Michael Flynn, Cambium Assessment, Inc.; Julie Benson, Cambium Assessment, Inc.; Kevin Dwyer, Cambium Assessment

• Organizational Adoption of Generative AI Assessment and Feedback Tools – Early Successes and Challenges - Rachel Forman, McKinsey & Company

Discussant

• Leslie Rutkowski, INDIANA UNIVERSITY