Assessing Reliability and Divergence of Human vs. Al Writing Evaluation in Chinese EFL Expository

• In Event: AI for Higher Education Assessment

Abstract

This study investigates the consistency and differences between human raters and ChatGPT in assessing Chinese EFL learners' expository writing. Eighty-two compositions were evaluated on language, content, and organization by humans and two ChatGPT sessions (ChatGPT1 and ChatGPT2). A repeated-measures two-way ANOVA and post-hoc t-tests analyzed rating patterns. Results show ChatGPT gave higher scores for content and organization, while humans rated language more favorably. No significant differences emerged between the two ChatGPT sessions, indicating strong internal consistency. The findings suggest AI can serve as a reliable complementary tool in writing assessment and support the development of hybrid evaluation models that combine human and AI strengths in second language writing.

Author

• Ruonan Yang, The Ohio State University