Inferring Emotional States During Metacomprehension Judgments: Evidence From Facial Expressions, Eye Movements, and Metacognitive Judgments

• In Event: 32.079 - What Can Be Inferred From Trace Data? Current Methods to Triangulate and Validate Traces of Learning Behavior

Abstract

Objectives: Metacomprehension is a significant contributor to problem-solving, reasoning, and academic achievement when learning complex science topics with multimedia materials (Azevedo, 2014; Dunlosky & Lipko, 2007). Unfortunately, learners do not always engage in accurate monitoring and regulation of comprehension that requires selecting, organizing, and integrating text and diagrams (Mayer, 2014). Literature has mainly focused on cognitive and metacognitive processes, without considering the role of emotions (e.g., confusion) that may be induced by cognitive factors (e.g., detecting discrepancies in multimedia content) and can affect the accuracy of metacognitive judgments (e.g., judgments of learning [JOLs]). Therefore, the goal of this paper is to explore whether facial data can contextualize metacognitive judgments made under different experimental conditions to improve our understanding of metacomprehension accuracy.
Theoretical Frameworks: We must combine multiple theories to explain the relationships between cognitive, emotional, and metacognitive processes during learning with multimedia as no single model addresses all processes. The levels of disruption hypothesis (Dunlosky & Rawson, 2005) and the Cue-Utilization Framework (Koriat, 1997) will be used to explain metacognitive judgments, while the CTML (Mayer, 2014) will be used to explain cognitive processes. Emotions will be explained by the Model of Affective Dynamics (D’Mello & Graesser, 2012) and the Resource Allocation Model (Ellis & Ashbrooke, 1988) will explain their influence on cognitive and metacognitive processes.
Method, Data, and Analyses: Thirty college students’ eye-movements, facial expressions of emotions, and metacognitive judgments were collected as they participated in a within-subjects study with multimedia science content containing conceptual discrepancies (i.e., none, text, and text and graph; based on Burkett & Azevedo, 2012). Participants’ eye-movements were analyzed by creating areas of interest (AOIs) on the text and graph using iMotions (2016) Attention Tool 6.1.1 (Figure 1). Videos of facial expressions were analyzed to detect and classify facial expressions of emotions. The software analyzes each video frame (30Hz) and provides evidence scores for each facial expression on a logarithmic (base 10) scale of an expression being present or not. Thus, positive evidence values for facial expressions of confusion were collected. Lastly, absolute accuracy scores for participants’ JOLs were calculated using Schraw’s (2009) absolute accuracy index.
Results indicate confusion was a significant moderator of the relationship between number of text fixations and text JOL accuracy, ΔR2 = .09, F(1, 29) = 4.37, p =.016. However, results indicate that confusion was not a significant moderator for graph fixations and graph JOL accuracy. These results indicate that emotions influence metacomprehension accuracy. Including data from facial expressions with eye-movements and logged events is a critical step in understanding the processes underlying successful self-regulated learning.
Significance: Despite the increase of online trace data collection, research still lacks in integrating multiple data channels (Figure 2) to explain the processes contributing to successful metacomprehension of multimedia (Azevedo et al., in press). This submission advances models of metamemory (Nelson & Narens, 1990) by examining multi-channel trace data in ecologically valid contexts. Future research addressing validity issues related to online trace methodologies is critical to enhancing our theoretical models of learning and instruction.

Authors

• Nicholas Vincent Mudrick, North Carolina State University

• Roger Azevedo, North Carolina State University

• Michelle Taub, North Carolina State University