Beyond Questionnaires: Measuring Self-Regulated Learning in LMS Using Multi-Channel Trace Data

• In Event: Authenticating the Signal: Validating Digital Traces of Student Learning Using Concurrent, Corroborating Data Sources

Abstract

Background & Aims. There is an ever-increasing use of technology across all educational sectors, resulting in tracking and understanding learning and teaching processes. The data generated by these educational systems offer an unprecedented opportunity to understand students’ behaviors at a level of analysis and scale that was not possible when bluebooks and multiple-choice exams were the primary evidence of learning. Traditionally, researchers have used questionnaires to measure different aspects of learning and instructions. Recently, several researchers have extended the theories by using observational data (e.g., log-files, eye tracking, physiological sensors, screen recordings of human-machine interactions, class- room discourse) to understand complex processes such as self-regulated learning (Azevedo et al. 2018). However, some researchers (e.g. Karabenick & Zusho, 2015), emphasized the importance of self-report and understanding students’ conception of themselves. On the other hand, other researchers (Greene & Azevedo, 2010) indicated that students are not accurate reporters of their behaviors, and therefore we should question the validity of self-reported data collected using surveys. In this study, we addressed this important methodological and theoretical debate in self-regulated learning research. We examined the extent to which self-report and observational data (i.e., LMS logs) in online learning converge.
Methods. This study consists of four main steps: First, we applied instructional design to create a template for a learning activity in the LMS to support students’ self-regulatory process and create visible and meaningful markers of student learning at different points in time in LMS logs. Second, we developed indicators of self-regulated learning phases from students’ trace data by applying the text replay tagging technique (Sao Pedro et al. 2013). Third, we developed an embedded tool in the LMS to collect students’ self-reported learning behaviors in real time. Figure 1 shows the prompt box (orange box), which asked students to report what they were doing on the assignment. Finally, we examined the convergence of multi-channel self-regulatory indicators using quadratic Cohen’s kappa coefficient. Our research design demonstrated how the triangulation of different sources of students’ self- regulatory data could help to unravel the complex nature of metacognitive processes.
Results & Significance. As shown in Table 1, our findings state how the indicators of self-regulation from two different channels were comparably converged in various learning tasks that targeted different cognitive skills, i.e., problem-solving and critical thinking. As Järvelä et al. (2019) indicated that there are limited methods to make mental regulation processes observable, we attempted in this study to find the proximal indicators of these invisible processes from two different channels, i.e., self-report and actual online behaviors. Although our understanding of these processes is yet inferential, multi-channel data can help us better picture these complex mental processes. Our indicators may help to provide more adaptive scaffolding for students to be effective self-regulated learners.

Authors

• Fatemeh Salehian Kia, University of British Columbia

• Ryan Shaun Baker, University of Pennsylvania

• Stephanie D. Teasley, University of Michigan