Tracing Students' Learning Behaviors in Large Face-to-Face Lecture Courses and Their Relations to Achievement

• In Event: Self-Regulated Learning Analytics: Aligning Data and Their Treatment to the Assumptions of Theory

Abstract

Objective & Theoretical Framework. Research examining learning management systems (LMSs) is limited, and mostly focuses on concerns such as usability and user satisfaction (e.g. Buzzetto-More & Sweat-Guy, 2006). Those who examine how student behaviors predict learning rely on coarse-grained indicators of behavior such as minutes using the LMS or frequency of access to predict outcomes (e.g. Crampton, Ragusa, & Cavanagh, 2012; Macfadyen, & Dawson, 2009). The handful of studies that utilize fine-grained measures of learning behavior demonstrate that behavioral logs compiled by LMSs are a rich, untapped resource for understanding learning (Black & Dawson, 2008; Romero, Ventura & Garcia, 2008). However, this research has been primarily conducted by computer scientists and educational data miners, who seldom apply a theoretical lens to understand learning. We present a project that leverages insight from self-regulated learning theory (Zimmerman & Schunk, 2011, Winne & Hadwin, 1998, 2008) and data mining techniques to understand how individuals use the course materials instructors provide via a LMS, and how they predict student outcomes.
Method. In collaboration with LMS administrators and STEM instructors, our team annotated source tables of the content provided by instructors by labeling content items and the kinds of learning behaviors they support (Table 1). We observed 549 undergraduates’ use of these resources when enrolled in Biology, Calculus, and Engineering courses and examined the associations of resource use with course performance (i.e. exams and course grade). Screenshots of these resources appear in Figure 1.

Results. While the resources provided to students differ by discipline and according to the learning objectives in each course, relationships between student use of resources and achievement in the course can be observed via correlations, regressions, and data mining methods. Sample findings include relations between the frequency of resource use and performance on the final exam and in the engineering course (Table 2), and prediction models using student resource use to predict performance in biology (Figure 2). In engineering, greater activity in the LMS environment predicted superior achievement, as did greater utilization of planning resources (i.e. syllabus & study guides) and lecture notes. By modeling biology students’ resource use, we predicted their performance at a rate better than chance by week 2 of the semester. In the session, we will present refined models and demonstrate how temporal information may further mediate the predictive effects of resource use on performance, and how sequences or combinations of behaviors may further inform predictions of performance.

Significance. Use of server logs to examine learning behaviors provides instructors with a precise record of students’ activities using unobtrusive observation methods that are already collected and monitored by university information technology departments (i.e. monitoring performances of the LMS server). When enriched with metadata about the nature of LMS content and timing of student use, these data can inform theories of self-regulated learning, enable deployment of feedback and support for learners, and allow teachers to adapt instruction.
With refinement, prediction models may be sufficiently accurate to identify students in need of learning support so that interventions can be delivered.

Authors

• Matthew L. Bernacki, University of Nevada - Las Vegas

• P. Merlin Uesbeck, University of Nevada - Las Vegas

• Nancy Webb, University of Nevada - Las Vegas

• Kyle B Bowen, University of Nevada - Las Vegas

• Lucie Vosicka, University of Nevada, Las Vegas

• Nicholas Diana, Carnegie Mellon University