Using Mixed Logistic Regression to Model the Language in Self-Regulated Learning Think-Aloud Codes

• In Event: Building Theory-Informed and Context-Sensitive Natural Language Processing to Capture Self-Regulated Learning in Action

Abstract

Purpose
Removing barriers to the use of high-quality methodology, like think-aloud protocols (TAPs), is a promising application of artificial intelligence (AI) methods. Though informative and valuable, TAPs also require a large investment in person-power and time (Hu & Gao, 2017), something that is not always cost-effective for researchers as they require multiple well-trained, internally reliable human coders. In this proposal, we leverage the strengths of ChatGPT to classify language from coded TAPs to investigate what categories of language help differentiate types of TAP codes from one another and take a step towards using Large Language Models (LLMs) to support researchers in using methods like TAPs. TAPs are a rich data source that can provide information on learners’ thought processes while they engage in a task and can measure complex phenomena, such as self-regulated learning (SRL; Lim et al., 2021). Through these insights into what categories of language are present and absent from groups of think-aloud codes, we develop both a better understanding of process-level SRL and generate materials to fine-tune an LLM to code TAP data more reliably than current models are able.

Framework
The TAP codebook and data used in this analysis are from prior work capturing learners' SRL process in large, undergraduate gateway courses (Authors, in press). SRL is defined as a cyclical process by which learners strategically engage their cognition, metacognition, motivation, and emotions in pursuit of a learning goal. We use an adapted version of this codebook, which identifies SRL processes, such as goal-setting and re-reading, and classifies them into groups based on process type (e.g., planning, strategy-use; Table 1).

Method
Nearly 6,000 coded text segments from 48 participants containing over 2,000 distinct terms were analyzed. We used ChatGPT4.0 and manual review to classify these terms into 23 categories (Table 2). Using a randomly assigned training set with a reserved test set, we trained a mixed logistic regression model (i.e., random effects for participants) to predict TAP code groups from term categories. During training, we used simple oversampling to make the target TAP code roughly 50% of the training set. The resulting models were then applied to the reserved test set of text segments (Table 3).

Results
Some results from the models were unsurprising and reaffirmed our understanding of the coding schema. For example, domain-specific TAP codes that exclusively related to math problem solving were most strongly predicted by mathematical operation language. The models also revealed interesting information about codes, such as metacognitive experiences being strongly predicted by negation language and metacognitive skills being strongly predicted by interjections (Table 4). In our final paper, we will discuss the nuances of these relationships in more detail.

Significance
Building an AI tool to reduce the human labor and time costs of TAP coding showed promise and can (1) improve multimodal modeling of SRL (Molenaar et al., 2023) via multimodal validation (i.e., digital-verbal) . (2) refine SRL TAP codebooks, and afford scaled TAP use to study SRL,.

Authors

• Christina Hollander-Blackmon, University of North Carolina - Chapel Hill

• Robert D Plumley, University of North Carolina - Chapel Hill

• Matthew L. Bernacki, University of North Carolina - Chapel Hill

• Jeff A. Greene, University of North Carolina - Chapel Hill