Analyzing Multimodal Data about Student Engagement: The Added Value of a Complex Dynamic Systems Approach

• In Event: Applying Complexity Frameworks to Self-Regulated Learning Research: From Theoretical Models to Practical Tools

Abstract

Objectives. There are three motives for analyzing multiple data about learning: (1) to promote a deep and accurate exploration of a learning phenomenon by integrating a broader range of related information (see Figure 1); (2) to enhance the credibility and trustworthiness of research findings by triangulating them with multimodal data (see Figure 2); and (3) to develop a holistic understanding of a learning construct by collecting and analyzing various data modalities on its subcomponents (see Figure 3). We introduce a complex dynamic systems (CDS) approach for analyzing multimodal data collected in learning or problem-solving (See Figure 4). We present a case study to illustrate how a CDS approach can be applied in the analysis of multimodal data (self-reports, eye gaze, head pose, and facial action units) to reveal student engagement. Specifically, we address the following research questions: (1) can eye gaze, head pose, and facial action units predict cognitive engagement? (2) does a complex dynamic systems (CDS) approach for analyzing eye gaze, head pose, and facial action units add value to the prediction analysis in the first question?; and (3) can a CDS approach provide additional insights into differences between high and low engagement levels?
Methods. The participants consisted of 61 medical students (52.5% males), who were tasked with diagnosing a virtual patient independently in BioWorld (see Figures 5-6), an intelligent tutoring system. An experience sampling method was used to measure students’ cognitive engagement in-situ during the problem-solving process, using the situational cognitive engagement instrument[14]. Furthermore, students’ head movement and facial behaviors were captured through video recordings using a webcam.
OpenFace 2.0 was used to analyze recorded videos and extract features from three data modalities: eye gaze, head pose, and facial action units[15] (See Table 1). The mean of self-reported cognitive engagement was calculated for each participant. Afterward, multiple linear regression was conducted where cognitive engagement was predicted from the continuous explanatory variables related to eye gaze, head pose, and facial action unit. We constructed a network of the explanatory variables for each participant, using a network estimation technique, EBICglasso (Extended Bayesian Information Criterion Graphical Least Absolute Shrinkage and Selection Operator). We calculated network density, incorporated it into the multiple regression model, and compared its performance with its previous iteration. We also constructed networks for low and high engagement groups.
Results. The multiple linear regression was statistically significant (Tables 2-4). Moreover, network density contributed significantly to the model (Tables 3 and 5). Table 6 indicates that network density was a more powerful indictor than other variables. Furthermore, there were 29 and 32 students in the low and high engagement group (Table 7). The network of the low engagement group exhibited isolated nodes, whereas the network of the high engagement group revealed specific interaction patterns between nodes (Figures 7-9).
Significance. This study proposed a CDS approach for analyzing multimodal data. The findings yielded insights about student engagement that cannot be obtained through the analysis of a single data modality.

Authors

• Shan Li, Lehigh University

• Juan Zheng, Lehigh University

• Susanne P. Lajoie, McGill University