Using Eye-Tracking and Log File Data as Indicators of Metacognitive Monitoring and Cognitive Learning Strategies With Game-Based Learning Environments?

• In Event: 32.079 - What Can Be Inferred From Trace Data? Current Methods to Triangulate and Validate Traces of Learning Behavior

Abstract

Objectives: Research on self-regulated learning (SRL) reveals students seldom deploy effective metacognitive monitoring and control strategies during learning (Azevedo et al., in press). Thus, advanced learning technologies (ALTs; e.g., game-based learning environments [GBLEs]) foster effective metacognitive processes (e.g., Azevedo et al., 2015). It is challenging to trace metacognitive monitoring using solely log files, as the majority of these processes are covert. As such, different trace data can be collected and converged to examine SRL. Few studies have validated measures of eye tracking to enrich log-file data to trace metacognitive monitoring. While the information processing theory (IPT; Winne & Hadwin, 2008) assesses online trace data to measure monitoring and control processes of SRL as an event that unfolds over time, it has yet to be tested with GBLEs. The goal of this study is to ensure validity with online trace data to examine the use monitoring and control processes with Crystal Island, a GBLE that requires solving the mystery of what disease impacted island inhabitants (Rowe et al., 2011).
Methods and Data Sources: We collected eye tracking and log files from 50 college students. To win Crystal Island, participants had to gather clues to solve the mystery by reading books and completing their associated assessments (concept matrices), interact with non-player characters, test food items, and document progress via a diagnosis worksheet (Figure 1). Data were run through a data pipeline (Figure 2), which calculated eye tracking (e.g., fixation duration) and log behaviors (e.g., selecting books), at the instance level (e.g., one interaction with that activity). We calculated the proportions of fixations on areas of interest for each instance.
Preliminary Results: Addressing the validity of converging trace data requires advanced statistics, such as multi-level modeling. Preliminary MLM results based on book reading with concept matrix instances indicated a significant 4-way interaction (ϒ41 = .077, t = 10.41, p < .0001). Participants who read fewer books, but read individual books more frequently (i.e., repeated access), and had low proportions of fixations on content and concept matrices performed the best (i.e., completed the concept matrices correctly in the fewest attempts). This suggests participants engaged in content evaluations (CE), a monitoring strategy where participants select relevant material to read (Greene & Azevedo, 2009). Although these results are inferential, this is one of the first studies that integrates eye tracking and log files to identify instances of monitoring and control, allowing us to situate this behavior within the IPT model.

Significance: Our results significantly augment the IPT model by integrating trace data from log files and eye tracking, providing evidence of metacognitive and control strategies. We address validity issues by converging trace data to provide evidence for the temporal dynamics of SRL during learning with GBLEs. We should continue combining methods (e.g., retrospective protocols) to provide additional validity for multi-channel data as a proxy for metacognitive monitoring. Ultimately, we aim to design adaptive ALTs (see Azevedo et al., in press) using multi-channel data to provide valid evidence that participants are engaging in monitoring and control strategies.

Authors

• Michelle Taub, North Carolina State University

• Nicholas Vincent Mudrick, North Carolina State University

• Roger Azevedo, North Carolina State University

• Garrett C. Millar, North Carolina State University

• Jonathan Rowe, North Carolina State University

• James Lester, North Carolina State University