A Cognitive Analysis Method for Assessing Self-Regulated Problem-Solving Processes

• In Event: Research Methodologies for Studying Problem-Based and Project-Based Learning

Abstract

Purpose
Problem-based learning (PBL) is a constructive process situated in authentic social contexts. It requires students to gain flexible knowledge, develop metacognitive reasoning, and collaborate on solutions (De Grave et al., 1996; Hmelo-Silver & Barrows, 2006; Smith et al., 2022). PBL necessitates a different research approach from the traditional one to capture students’ knowledge construction and problem solving over time (Eseryel et al., 2011). To accomplish the goal of PBL research, we need powerful tools to accurately describe learners’ cognitive processes because ultimately PBL focuses on reasoning, problem solving and metacognition (De Grave et al., 1996; Du et al., 2009). The purpose of this research presents a cognitive analysis method to assess students’ self-regulated problem solving.

Theoretical Perspective
From the constructivist learning perspective, PBL is intended to engage students in knowledge construction and building and create solutions for real-world problems (Hmelo-Silver & Barrows, 2006; Savery, 2006). Therefore, PBL assessment should capture students’ learning processes and experiences, with artifacts as indicators of learning progress and achievements. PBL and self-regulated learning are intertwined and contextualized, which presents a challenge for assessment (Winne, 2010). Learning experiences (information processed and explored) feeds into the learning context, which in turn unfolds the learning process. As a result, PBL researchers need a research method that captures PBL processes and themes.

Method
Chi (1997) and Ericsson and Simon (1980) developed practicable procedures for protocol analysis, which have been shown as a valid and reliable method to assess problem solving (Tarciani & Clariana, 2006). However, there is a lack of a systematic research method for analyzing think-aloud data in the PBL context. Drawing from Miles and Huberman’s work (1994) of data analysis, visual display, and organization, and based on our research experiences, we generated a model of five steps for analyzing self-regulated PBL processes:
1. Conceptualizing self-regulated problem-solving processes grounded in a theoretical framework that guide the cognitive analysis
2. Administering think-aloud protocol
3. Within-case analysis: Coding, visual displaying, and organization
4. Cross-case analysis: Comparing cases organized with a visual display
5. Examining patterns and drawing themes, triangulated with other sources of data to build theories.

Data and Evidence
The cognitive analysis method has been empirically proven as insightful for examining students’ cognitive and metacognitive processes during PBL (Authors, 2003; and Authors, 2019). In this symposium, we will share three studies where cognitive analysis was adapted: (1) students’ ill-structured problem-solving processes prompted by the question prompt strategy (Authors, 2003), (2) students’ iterative information search behaviors triggered by various factors (Authors, 2019), (3) and self-regulated problem solving during a gameplay. In all these examples, both within-case and cross-case analyses were conducted to provide evidence of students’ reasoning processes.

Significance
The cognitive analysis provides a deeper insight into students’ reasoning, decision-making and problem-solving processes in PBL environments. It equips educators and researchers with a conceptual tool to better diagnose students’ challenges and understand their PBL experiences and performance so that appropriate and just-in-time scaffolding can be provided. Future effort will focus on developing an automated cognitive analysis system.

Authors

• Xun Ge, University of North Texas

• Victor Law, University of New Mexico

• Kun Huang, University of Kentucky