Structured Transactional Event Analysis of Multimodal Streams (STREAMS): A Tool for Validating Trace-Based Inferences About Learning Behavior

• In Event: 32.079 - What Can Be Inferred From Trace Data? Current Methods to Triangulate and Validate Traces of Learning Behavior

Abstract

Objectives and Perspectives: Due to the relative difficulty of collecting multi-modal, context-rich data during technology-supported learning, research has largely focused on making inferences about student learning processes from traces that educational technologies log. Educational research rarely moves beyond these traces for a number of reasons. Data on classroom context are difficult to collect, and data from different sources are often collected at different grain sizes, which are difficult to integrate. Self-regulated learning is known to unfold in context – the appropriateness of learners’ actions is contingent upon the events that proceed them and task conditions (Winne & Hadwin, 2008). In order to validate our inferences about the SRL behaviors traced in logs, we developed a methodological pipeline that facilitates the collection, synchronization, and analysis of multi-modal, context-rich data streams.
Methods: Additional streams of contextually rich data can be collected using cameras, microphones, eye-trackers, sensors and other tools. We aimed to minimize both deployment effort and interference with students’ learning experience by using Camtasia, which runs on students’ computers to collect audio, screen video, and student-facing webcam data while they engage with educational technology.
We developed a tool called Structured TRansactional Event Analysis of Multiple Streams (STREAMS) that links computer-logged trace data with corresponding audio-video streams and automatically extract audio-video epochs associated with specific events of interest. The tool allows researchers to indicate any event type that is tagged within the log data. It then generates a folder of clips containing all epochs of the synchronized audio-video data streams pertaining to events of interest. With the relevant epochs grouped together to allow for efficient analysis by a human examiner, it becomes easy to use multi-modal data streams to validate inferences about student learning that are made from trace data.
We applied this method by deploying Camtasia alongside two classroom studies examining (1) students working individually on a Chemistry Virtual Lab tutor, and (2) collaboratively on a Fractions Tutor.
Results: We used our tool to examine multi-modal streams during events where we felt that additional context would help us better understand learning behaviors. For example, we identified certain periods of learning within the collaborative Fractions Tutor during which the trace data revealed accelerated progress. However, it was not clear from the trace data why the students were working faster. Using the STREAMS tool, we extracted epochs from the relevant moments and observed that pairs of students were comparing their progress to other student-pairs during these moments. The audio/video data streams revealed a spirit of competitiveness in the dialogue between students that led to the accelerated pace of learning during these moments.
Significance: This result is just one of many examples of ways in which additional context-rich data streams may be used in ways that can both supplement and validate inferences made from trace data. The STREAMS tool may prove useful in a variety of other ways, including providing ground truth labels for error attribution, validating the relative contributions of individual partners during collaborative learning, and making inferences about affective states during learning.

Authors

• Ran Liu, Carnegie Mellon University

• John Stamper, Carnegie Mellon University