Exploring Critical Engagement with GAI in Academic Writing: A Mixed-Methods Study

• In Event: AERA Poster Session Saturday 9:45 am
  In Poster Session: AI and Technology Integration in K-12 Education

Abstract

This mixed-methods study investigates undergraduate students’ critical engagement with ChatGPT-4o during academic writing revision and its impact on writing performance. Drawing on an adapted Critical Interaction with AI for Writing framework, we analyzed 102 prompts from 41 students across four critical engagement dimensions: content revision, information seeking, writing presentation, and conversational engagement. Results of thematic analysis found that most students engaged at shallow to moderate levels, with limited critical evaluation on AI responses. Linear mixed-effects modeling revealed that only content-focused critical engagement significantly predicted improvement in idea presentation of writing. These findings urgent need for further pedagogical scaffolding that promotes critical, reflective, and self-regulated use of GAI tools to enhance academic writing and higher-order thinking development.

Authors

• Qi Sun, University at Albany - SUNY

• Shengkai Yin, Federation University Australia