Modeling Rater Judgments on AI-Aided Rubrics: A Rasch-Based Mixed Methods Study

• In Event: Rasch Model Applications including AI

Abstract

This study used Many-Facet Rasch Measurement (MFRM) to evaluate the quality of analytic rubrics developed with and without Artificial Intelligence (AI) support. Sixty rubrics (30 AI-assisted, 30 non-AI) were rated by ten expert raters across five domains. MFRM revealed substantial differences in perceived rubric quality and rater behavior across conditions. To support interpretation, qualitative reflections from rubric authors were analyzed, revealing alignment with model findings and offering insight into how AI influenced rubric structure and clarity. This embedded mixed methods design highlights the value of MFRM in detecting nuanced differences in performance task design quality.

Authors

• Beyza Aksu-Dunya, Bartin University

• Talip Gülle, Jyvaskyla University

• Mehmet Can Demir