Challenges in Using Data Mining to Identify Robust Indicators of Cognitive, Affective, and Metacognitive Self-Regulatory Processes From Trace Data During Learning With Advanced Learning Technologies

• In Event: 44.081 - Data-Driven Techniques That Complement Theoretical Study of Cognition and Learning With Technology

Abstract

Objectives. Data mining continues to generate deep insights into SRL (Winne & Baker, 2013), but key theoretical, analytical, and educational issues persist (Authors, Date). We address two issues—(1) identifying robust indicators of cognitive, metacognitive, and affective SRL processes from multichannel data (e.g., log files, eye tracking, physiological sensors) during learning with an intelligent tutoring system (ITS) for biology (MetaTutor; Authors, Date); and (2) illustrating how features of multichannel data (e.g., frequency and quality of strategy use over time) can be examined and modeled using data mining.
Methods. We tested whether external regulation provided by pedagogical agents (PAs) was effective in facilitating SRL with MetaTutor. 170 undergraduates learned about the circulatory system with MetaTutor during a 2-hour session under one of two conditions: adaptive scaffolding (AS) or a control (C) condition. The AS condition received prompts from four PAs to deploy cognitive and metacognitive SRL processes, and received immediate directive feedback concerning the deployment of the processes. By contrast, the C condition learned without assistance from the PAs. Results indicated that those in the AS condition learned significantly more about the science topic than those in the C condition. During the learning session, we collected the following multichannel data from each participant: log files, eye tracking, video recordings of the face, SRL measures (e.g., quiz results, summaries and metacognitive judgments, PA feedback), and electrodermal activity. We also collected pretest and posttest data and several self-report measures on emotions, motivation, and agent likeability and metacognitive knowledge about specific SRL processes.
Results. The multichannel data involving individual learners’ multidimensional traits were used for learner modeling through latent class analysis (LCA) (Lazarsfeld & Henry, 1968). Our evidentiary reasoning from the LCA-based learning modeling was focused on configuring heterogeneity in learner profiles resulting from intrapersonal (cognitive, metacognitive, affective) and interpersonal (treatment vs. condition) differences. Seven SRL behavior variables identified three distinct SRL classes that best fit the data based on model fit criteria, such as AIC, BIC, and Entropy (classification accuracy). Figure 1 shows the probability of class membership across seven SRL behavior variables.
Subsequent logistic regression analyses showed that the high SRL class (Class 1) tended to complete more subgoals (p = .001), more quizzes (p = .000), and achieved higher gain scores (p = .031). We further examined whether there is any difference in achievement among three SRL classes. Figure 2 presents a scatterplot of pretest by posttest scores across three SRL classes.
We further examined the relationship between SRL classes and achievement patterns by classifying students into nine total composite memberships (low/high start vs. low/high gain vs. low/mid/high SRL). Figure 3 shows three (out of nine) composite profiles. The composite class 1 (yellow) represents students who showed large gain scores while belonging to SRL class 1. This class tended to show higher emotional engagement and interest in their performance.

Authors

• Roger Azevedo, North Carolina State University

• Joseph Grafsgaard, North Carolina State University

• Michelle Taub, North Carolina State University

• Nicholas Vincent Mudrick, North Carolina State University

• Eunice Eunhee Jang, University of Toronto

• Clarissa Lau, University of Toronto

• Jeanne Sinclair, University of Toronto