How Well Do Logged Events Represent Students' Learning Processes? Aligning Students' Reports With Trace Data

• In Event: 32.079 - What Can Be Inferred From Trace Data? Current Methods to Triangulate and Validate Traces of Learning Behavior

Abstract

Purpose & Theoretical Framework: Trace data capture self-regulated learning (SRL) events that predict performance (Bernacki, Byrnes, & Cromley, 2012) and determine feedback learners receive (Dominguez & Bernacki, 2016). Learning management system (LMS) sites frequently supplement face-to-face courses and trace use of resources including syllabi, lecture notes, practice quizzes and study guides. While the occurrence of a traced event is certain, the student’s intentions are not. Events are coarse data that may not precisely trace SRL microprocesses when used individually (Greene & Azevedo, 2009) and must be carefully modeled to represent more dynamic and contingent SRL processes (Winne, 2011).
LMS events are ubiquitous, practical data, but the validity of inferences that can be drawn from them remains in question. This study assessed the construct validity of traced events and investigated how students’ self-reports of the purpose align to the SRL processes resources were designed to support. Analyses examined (1) the homo/heterogeneity of student’s purposes for using resources to determine whether use represents one or many possible SRL processes and (2) whether more complex treatment of logged events can provide theoretically aligned, precise traces that disentangle the ways students describe their resource use.
Data: Self-report data and logs of learning behaviors were obtained from undergraduates in two semesters of an Anatomy and Physiology lecture course. Sample 1 (N=325) completed open-ended prompts to describe use of LMS-hosted resources. Sample 2 (N=158) endorsed multiple-response options to describe reasons for resource use. Behavioral data consist of time-stamped logs of students’ use of syllabi, lecture notes, learning objectives (with self-rating tool to assess mastery), practice quizzes (enabling retrieval practice and self-assessment), exam study guides, and a page displaying course grades.
Methods: Analyses used open- and close-ended responses to identify whether resource use reflects one or many SRL processes. Open responses were analyzed qualitatively to identify reasons for use. Frequencies of endorsement of purpose(s) were obtained from close-ended questions. Analysis and trace construction that align self-reports to logs of resource use are ongoing.
Results: In both samples, students endorsed multiple rationales for using each resource (Tables 1, 2). Endorsement patterns in close-ended items indicate a primary use for some resources, but heterogenous endorsement was also observed for all resources. Results indicate that logged uses of resources may be insufficient evidence to infer occurrence of a specific SRL process, and that confidence in inferences should vary by the resource used. Additional analyses will demonstrate how events’ metadata and patterns, co-occurrences, or sequences of events might serve as more valid traces of SRL microprocesses (Figure 1).
Scholarly significance: LMS log data provide evidence of students’ behaviors, but drawing inferences solely from event logs is problematic. Descriptions of learning that rely on logs alone should qualify inferences accordingly, and researchers who wish to test research questions involving specific SRL processes must work to increase the likelihood that logged events truly reflect focal (meta)cognitive processes. Methods might include training resource use for a sole purpose, or development of more complex traces built on multiple events/features to precisely trace learning processes.

Authors

• Lucie Vosicka, University of Nevada, Las Vegas

• Matthew L. Bernacki, University of Nevada - Las Vegas