Color me confounded: A critical analysis of media comparisons on ChatGPT in education

• In Event: AERA Poster Session Wednesday 9:45am
  In Poster Session: Diverse Learners and Inclusive Teaching Practices

Abstract

Recent meta-analyses were published (Deng et al., 2025; Wang & Fan, 2025) that examined media comparisons between ChatGPT and “traditional learning”; both reviews found ChatGPT improved learning. However, instructional methods are often not controlled between conditions, making it difficult to attribute any learning differences to the medium itself. Our critical review examined those media comparisons studies included in Deng et al. and Wang and Fan to determine whether the ChatGPT and control conditions were comparable on their instructional features. Results indicated a lack of control in ChatGPT studies, with the majority of studies conflating the use of ChatGPT with other instructional features. Therefore, more research is needed to determine how, for whom, and under what conditions ChatGPT can improve learning.

Authors

• Alyssa Lawson, Colby College

• Amédee Marchand Martella, University of Georgia

• Joshua Weidlich, Heidelberg University of Education

• Miriam Mulders, University of Duisburg-Essen

• Josef Buchner, St. Gallen University of Teacher Education