Concurrent Think-Alouds: Using Trace Data to Understand Self-Regulated Learning

• In Event: 32.079 - What Can Be Inferred From Trace Data? Current Methods to Triangulate and Validate Traces of Learning Behavior

Abstract

Objective & Theoretical Framework: Emerging technology-based learning contexts such as flipped classroom and educational social media provide unique opportunities for students to learn in the order and representation of their choosing. Because learning in these environments is inherently nonlinear and resources are complex, students must self-regulate their learning process (Azevedo, Taub & Mudrick, 2015; Shapiro, 2008). Historically, researchers have relied on students’ self-reports of their behaviors to understand SRL processes, but the validity of these measures has been questioned because of problems with the precision of aggregated, retrospective reporting (Azevedo, 2014; Cleary & Platten, 2013; Greene et al. 2015; Hadwin et al. 2010; Winne & Perry, 2000; Winne & Azevedo, 2014). As a result, researchers have increasingly come to rely upon concurrent tracing of learning events, such as the think-aloud.
Methods & Data Sources: The think aloud has an extensive history in cognitive psychology and cognitive science (see Ericsson, 2006 and Ericsson & Simon, 1994 for reviews). Verbalizing thoughts during learning will not disrupt the learning process, if the “…subjects verbalizing their thoughts while performing a task do not describe or explain what they are doing” (Ericsson & Simon, 1994, pg. xiii). As such, the think-aloud protocol has the potential to produce valid trace data on students’ cognitive and metacognitive activity during a learning task. While currently the most established approach, the concurrent think aloud data is not without issue and can benefit from cross validation with other forms of trace data for various reasons. For example, asking adult learners to think-aloud does not seem to disrupt their learning, as evidenced by cognitive load data (Moos & Bonde, 2015). However, younger developmental groups may experience additional mental load when asked to think-aloud due to their developing expertise related to SRL. Furthermore, the think-aloud may be less effective in eliciting verbalizations related to specific aspects of self-regulation. In particular, the think-aloud has not demonstrated an ability to capture motivation events during learning. In the absence of data from motivation measures, the dynamic interaction between specific cognitive and metacognitive processes cannot be fully explained. As a result, research has turned to additional methods for measuring these specific SRL processes (Bernacki, Aleven, & Nokes-Malach, 2014; Bernacki, Nokes-Malach, & Aleven, 2015; Moos & Azevedo, 2008). This presentation will unpack the methodology of concurrent think-alouds, address potential challenges, and describe how our use of this measurement tool has evolved over the last 10 years.
Results & Significance: This presentation will use our empirical results and experiences to address important theoretical, methodological, and practical questions surrounding concurrent think-alouds, including: (1) What additional sources of data should be used to better understand the dynamic relationship between cognitive and metacognitive processes (e.g., motivation measures)?; and (2) How can a coding scheme be applied to data generated by think-alouds that reflects theoretical assumptions and the current state of SRL research (e.g., coding judgment of learning valences)?

Author

• Daniel Charles Moos, Gustavus Adolphus College