Multimodal Participation Analysis in CSCL Through AI-Enhanced Activity Mapping and Screen Recording Video Analysis Systems

• In Event: AERA Poster Session Wednesday 11:45am
  In Poster Session: ATL Poster Session 1: AI Integration and Student Outcomes in K–12 Settings

Abstract

This study presents a novel mixed-methods approach to analyzing multimodal participation in Computer-Supported Collaborative Learning (CSCL). Focusing on typing as a key form of object-based participation, we introduce two tools: an AI-based Activity Mapping Video Analysis system that analyzes collaborative group interaction video using neural network models, and a Screen Recording Video Analysis system leveraging Optical Character Recognition. Applied to a collaborative robotics programming session among middle school students, these systems enabled detection of who typed, when, and what was typed. Integrated with qualitative video-based interaction analysis, the findings reveal dynamic shifts in participation, tool access, and role negotiation. This approach demonstrates how AI can augment, rather than replace, human interpretation to support scalable, context-sensitive insights into real-world CSCL processes.

Authors

• Hakeoung Hannah Lee, University of Virginia

• Miguel E Lujan, University of New Mexico

• Venkatesh Jatla, Virginia Tech Carilion

• Ugesh Egala, Independent Scholar

• Marios S. Pattichis, University of New Mexico

• Sylvia Celedon-Pattichis, University of Texas at Austin

• Esteban Cantu, University of Texas at Austin