AI-Assisted Analysis of Peer Interaction in Higher Education Courses (Poster 14)

• In Event: AERA Poster Session Thursday 3:35 p.m.
  In Poster Session: (Generative) Artificial Intelligence in Education

Abstract

This study investigates the effectiveness of AI in replicating human coding of student interactions in postsecondary courses. Two prompting methods were employed: "chain of thought" (CoT) and "chain of thought with example" (CoTE). ChatGPT-4o compared each method against human coding across six classrooms of introductory statistics. Agreement was assessed with intraclass correlation coefficients (ICC), and coding discrepancies were visualized with Bland-Altman plots. The findings indicate that CoTE, which includes contextual examples, achieved higher ICC values, indicating better AI accuracy in analyzing peer interaction. The findings also revealed greater coding agreement between humans and AI in lecture-based classrooms, but less agreement when students were more active. This study highlights AI’s potential role in classroom observations and the need for further research.

Authors

• Vanessa Peters Hinton, Digital Promise

• Xin Wei, Digital Promise