Examining Undergraduate Students’ Thinking Processes and Programming Performance When Using Web Search and Generative AI

• In Event: AERA Roundtable Session Sunday 1:45 pm - Gold 3
  In Roundtable Session: AI in Student Learning: Programming, Writing, and Engagement

Abstract

This study examines the impact of generative AI (i.e., ChatGPT) versus traditional web search (i.e., Google search) on students’ thinking processes and programming performance. We compared two student groups using ChatGPT and Google during programming tasks, respectively. Markov chains revealed that the ChatGPT group exhibited more revision-oriented thinking processes, indicated by frequent transitions between question formulation and solution refinement. In contrast, the control group was more solution-development-oriented. ANCOVA results showed that ChatGPT-supported learners performed significantly better in programming tasks. These findings suggest that generative AI enhances performance by supporting rapid idea testing and refinement, while Google involves more self-directed solution development, potentially slowing progress. The study contributes to the understanding of how external tools impact thinking processes in programming tasks.

Authors

• Tianlong Zhong, Nanyang Technological University

• Gaoxia Zhu, National Institute of Education - Nanyang Technological University