Integrating Generative AI in Qualitative Analysis: Application in Science Education

• In Event: AERA Poster Session Friday 7:45 am
  In Poster Session: ATL Poster Session 2: AI Integration and Outcomes in Postsecondary Settings

Abstract

This study investigates the use of a multimodal large language model, Claude 3.5, in analyzing middle school students’ scientific models. We examine Claude’s performance in two tasks: (1) segmenting student responses into idea units and (2) coding them for correctness and reasoning type. Data includes models written and drawn by students during their science class. Findings show moderate agreement between Claude and human coders (Cohen’s κ = 0.43), with strong alignment on ideas coded as scientifically correct. This suggests the potential of generative AI to support fine-grained qualitative analysis and surface new insights into the composition of student thinking. We discuss implications for scaling the data analysis and strengthening human-AI collaboration in learning sciences research.

Authors

• Rida Munir, Utah State University

• Ha Nguyen, University of North Carolina - Chapel Hill

• Hillary Lucille Swanson, Utah State University

• Ravi Sinha, Utah State University

• Idris Solola, Utah State University