Two Methods of Extracting Campus Data, Interpreting Learning Management System Events That Reflect Learning, and Predicting Achievement

• In Event: Data Mining and Self-Regulated Learning: Aligning Constructs With Measurement

Abstract

Purpose and Theoretical Framework
Workforce demands put universities under increasing pressure to retain students, ensure they maintain progress towards degrees, and graduate students on time. Accordingly, university offices responsible for student success aim to develop ways of identifying students at risk of poor outcomes in courses. While many choose to purchase licensed access to early warning system that may or may not be attuned to their campus context, an emerging trend is to leverage one’s own campus experts and data that are locally available to develop systems to identify students who are candidates for learning support (Romero & Ventura, 2013). This paper describes two popular methods of extracting data on student learning from the campus learning management system where students learn with digital media: traces derived from server logs and from tables in a data lake. We demonstrate how the same learning events can be gathered in each method to represent key events reflecting self-regulated learning (SRL; Bernacki, 2018), and associated with student achievement in STEM courses as a preliminary step towards developing the prediction algorithms at the core of early warning systems (Bernacki, 2019).

Method
We analyzed two introductory Biology courses hosted in different LMSs: Blackboard Learn (2017) and Canvas (2019; n1=345, n2=391). Student activity on Blackboard Learn was extracted from server logs. Parallel events in Canvas were queried from a relational database (i.e., Oracle). Metadata from logs provided learning object names, which were aggregated to reflect use of content designed to support specific learning processes (Table 1). Canvas events were queried from many tables and merged to reconstitute a log-like record of learning events and handled similarly. We provide yoked examples of these methodologies and demonstrate how traces derived via each method can be extracted to represent learning events and associated with learning outcomes to inform “feature selection” processes when developing prediction models (Biswas, Baker, Paquette, 2018).

Results
Analyses examining relations among behavioral data reflecting learning events in Blackboard- and Canvas-hosted biology courses and three achievement indicators demonstrate that both extraction methods produce traces that correlated significantly with learning. Though instructional design features differed across courses, multiple learning resources were provided to students across LMSs, and use of them was consistently related to academic achievement. Navigation, access, and download of digital learning objects (e..g, course lecture note) was consistently associated with their achievement, as were use of resources reflecting metacognitive planning, monitoring and evaluation processes (e.g., Planning with study guides, self-quizzing [which was more predictive when required than self-initiated], use of homework solutions, and monitoring performance using the gradebook).

Significance
Campuses can choose multiple ways of extracting learning events that can predict student achievement and identify students likely to perform poorly in critical courses. Collaborations with educational researchers to refine and interpret the traces they extract can produce more precise prediction models and interventions designed to respond to the difficulties demonstrated by students’ learning behavior.

Authors

• Wonjoon Hong, University of Nevada - Las Vegas

• Matthew L. Bernacki, University of North Carolina Chapel Hill