BackHome
Refresh: OffView Personal Schedule
Paper Summary
Share...

Direct link:

Code What I Say, Not Just What I Do! Multimethod Investigations Into Digitally Traced Behaviors

Fri, April 22, 11:30am to 1:00pm PDT (11:30am to 1:00pm PDT), Marriott Marquis San Diego Marina, Floor: North Building, Lobby Level, Marriott Grand Ballroom 9

Abstract

Background. Researchers have struggled to capture detailed accounts of students’ self-regulated learning (SRL) in naturalistic settings. Early researchers relied on survey approaches, but respondents struggled to recall and report accurately when surveyed retrospectively (Winne & Jamieson-Noel, 2002). Research using think aloud protocols provided timely reports of students’ cognitive, metacognitive, and motivational processes during learning (e.g., Greene et al., 2010), but data collection and processing were effortful, and verbal traces could not be collected at a scale necessary to test complex assumptions from SRL models (Ben-Eliyahu & Bernacki, 2015). Growth in learning analytics has led to wide adoption of research involving digital traces of learning events (Winne, 2020). However, these are limited to events collected from server logs, with no corroboration by learners about what SRL process an event might reflect.
Aim. We sampled an authentic task from an undergraduate course, replicated it in the lab, and recruited enrolled students to complete it. We simultaneously traced students’ digital learning events (Bernacki, 2018) while collecting their verbalizations using think-aloud protocols (Greene et al., 2018) to examine how verbal and digital events co-occurred, and whether verbalized SRL processes could be validly inferred from reliable co-occurrence with digital traces.
Methods & Data. Undergraduate biology students (N=48) completed a lesson on evolution sampled from the last unit of their introductory course prior to coverage in class. They completed all phases of their typical “learning cycle” in their high-structure lecture course (see Figure 1), verbalized their thinking while completing preparatory, lecture, and post-assessment phases. Verbalizations were coded according to an established and augmented codebook for SRL in science learning contexts (see Table 1; Percent agreement > 75%)
Findings. The 48 learners produced 1,211 digital traces and verbalized 4,161 utterances that received an SRL code; 405 digital traces overlapped a verbal trace, and provide the opportunity to inspect the homogeneity vs. heterogeneity of SRL processes that learners might be engaging when they produce a digital trace during learning. We examined the frequency of digital and verbal traces over learning phases (Figure 1), then the co-occurrence of events descriptively using a cross-tabular method that aligned the digital trace set that was captured against the coded verbalizations that reflected micro-level and macro-level SRL processes (Figure 2). The frequency of instances where a verbalization overlapped with a digital event was tallied in a cell, and the cells were heat-mapped to show homogeneity (columns with a singular, brightly-hued cell) vs. heterogeneity. Inferential testing using chi-squared analyses for each digital trace confirmed that distributions of co-occurring verbalizations differed significantly from chance (see * and ^ in Figure 2). This provides confidence that specific SRL processes can be reliably inferred when digital learning events are recorded.
Significance. This corroboration of digital learning events that reliably co-occur with a single, verbal trace of a SRL process provide validity evidence for the inferences researchers can make about learning events observed in learning analytics studies of authentic learning contexts. These digital events can be scaled and submitted to analyses to test complex assumptions about SRL.

Authors