Impact of a GenAI-assisted Socratic Dialogue Tool on Learning in an Asynchronous Computer Science Course

• In Event: AI-Mediated Learning Environments: Scaffolding Inquiry, Metacognition, and Engagement

Abstract

This study examines the impact of a GenAI-powered formative assessment tool that employs Socratic questioning to support student learning in a large, fully online undergraduate-level computing course. Using a quasi-experimental, mixed-methods design, we investigated participants’ engagement patterns, the influence of user experience on engagement, and impacts on both perceived and actual learning outcomes. Survey data, system logs, and academic performance measures revealed strong engagement and user satisfaction, with students reporting perceived gains in higher-order thinking and overall learning outcomes. Students who engaged with the GenAI tool experienced significant gains in their quiz scores, particularly benefiting students with lower baseline achievement. Our findings highlight the promise of AI-mediated dialogue in fostering deeper engagement as a scalable instructional strategy.

Authors

• Jeonghyun Lee, Georgia Institute of Technology

• Meryem Yilmaz Soylu, Georgia Institute of Technology

• Jui-Tse Hung, Georgia Institute of Technology

• Diana Popescu, Georgia Institute of Technology

• Christopher Cui, University of California - San Diego

• Gayane Grigoryan, Georgia Institute of Technology

• David Joyner, Georgia Institute of Technology

• Stephen W Harmon, Georgia Institute of Technology