Data-Driven Digital Alerts and Learning Support: Effects on Achievement and Moderation by Unintended Course Events

• In Event: Scaffolding Self-Regulation, Co-Regulation, and Socially Shared Regulation of Future Learning: Affordances of Learning Analytics Dashboards

Abstract

Universities continually seek innovative methods to support STEM learners in challenging courses with high attrition and failure rates. The emergence of technologies that log student activity and analytic tools that can process these data to provide insight on learning processes afford new opportunities to deliver feedback and support to learners (Verbert, Duval, Klerkx, Govaerts & Santos, 2013).
We utilized trace data created by students when they interact with digital course resources hosted on the learning management system (LMS) site of a calculus course. Data on student activity and achievement (N=507) from multiple sections taught by two instructors and a forward selection logistic regression algorithm (with 10-fold cross validation) were used to train and test a prediction model that accurately identified >75% of students unlikely to achieve a B or better by the 4th week of the course. Accuracy held across instructors in test data and a subsequent semester, and was thus used to deploy an early alert system. Students predicted to perform poorly received an email on day 1 of week 4, a week before their first exam. It came from their instructor and linked to advice from successful students and learning skills training (Figure 1). Authors (date; N=199) found students who received messages and accessed learning supports outperformed those who received no message and did not access materials on subsequent exams (ds .5 to .8). We sought to replicate this effect in additional course sections to examine stability of the prediction model and effects of intervention as we consider scalability.
Trace data from 147 students’ LMS use generated real-time predictions in Splunk software (Authors, Date). Students enrolled in Spring 2017 sections were taught by the same two instructors with identical course content, exams, and schedule. On day 1 of week 4, 160 students were predicted to perform poorly, and 80 were randomly selected and messaged.
Student responsiveness to messaging declined from prior study (Table 1), and effects of messaging differed starkly across course sections. Messaged students in section A outperformed unmessaged students across 5 exams (ds = .09 to 1.107) but unmessaged students outperformed messaged students in Section B (ds = .01 to -2.53; Table 2, Figure 2). Exploratory analyses into LMS activity confirm that unplanned occurrences may have caused these strikingly different effects. Intervention fidelity in Section A mirrored the Fall 2016 implementation and the instructor sent an additional message announcing a first calculation of homework grades was posted in the LMS. In Section B, the instructor fell ill and announced that class was cancelled hours after students received the alert message. Logs confirm significant differences in Week 4 LMS activity across sections (Figure 3), potentially induced by contrasting instructor announcements. These occurrences may be responsible for effects on achievement. We thus conclude that digital, data-driven resources have potential to support learners, but designers must monitor impact of environmental factors and collaborate with instructors to ensure implementation fidelity and increase likelihood of anticipated effects.

Authors

• Wonjoon Hong, University of Nevada - Las Vegas

• Matthew L. Bernacki, University of Nevada - Las Vegas