Can We Trust AI as a Score Rater? A Preliminary Study on the Reliability and Validity of a GenAI Model’s Assessment of Learning Performance Data

• In Event: AERA Roundtable Session Sunday 7:45 am - Gold 4
  In Roundtable Session: VR and Haptic Learning Experiences

Abstract

This preliminary study explored the reliability and validity of Google Gemini 2.5 Pro, a Generative AI (GenAI) model, in assessing learning performance. We compared AI-generated scores with automated ratings given by the VR system for 77 nursing students' transcripts from a virtual reality patient encounter training using a 15-point rubric. Two rounds of AI scoring were conducted with different prompts. While average scores were comparable across raters (ANOVA, p > 0.05), inter-rater reliability between the VR system and AI was low (Cohen’s Kappa: 0.20 and 0.37). Criterion-related validity was moderate (Pearson's r: 0.65 and 0.77). Notably, prompting the AI to provide a detailed rationale improved both reliability and validity, suggesting that prompt engineering is crucial for enhancing GenAI's assessment accuracy.

Authors

• Hao He, Emporia State University

• Xinhao Xu, University of Missouri

• Jhon Bueno-Vesga, Pennsylvania State University

• Yupei Duan, University of Missouri

• Shangman Li, University of Missouri

• Yuanyuan Gu, University of Missouri

• Sue Yun Fowler, University of Missouri

• Hillary L. Claunch, University of Missouri

• Jason Snyder, University of Missouri