A Method for Representing Cognitive Processes in Think-aloud Protocols

• In Event: On the Comparability of Cross-National Assessment of Critical Thinking

Abstract

1. Objectives or Purposes
Think-aloud verbal protocols validly capture cognitive (thinking) processes (Ericsson, 2017). Typically, judges score verbal protocols by counting or rating variables representing the construct of interest. The result is a static representation of an individual’s thinking. Yet, that thinking occurs sequentially with transitions as a task is completed. Flow-charting, the focus of this study, is one possible method for tracing thinking processes. We present a method for deriving flow charts and quantitative indicators from the verbal protocols.
2. Theoretical Framework
The think-aloud method was initially used to capture cognitive processes in problem-solving (Newell & Simon, 1959; Ericsson, 2017; Leighton, 2017; in teaching Shavelson, Webb & Burstein, 1976). The sequence of verbal utterances reflects the thinking processes from the beginning to the end of the task solution. The transition from one utterance (e.g., idea, claim, decision) to the next, ordered in time, reflects cognitive processing. To capture the process rather than the static nature of critical thinking (CT), we use flow charts that time tag utterances and connect them over time (Fig. 1; process tracing—Shavelson et al., 1976). Flow charts provide sequential, qualitative, and quantitative information that characterize critical thinking.
3. Method
The flow-chart method was developed to capture the response process and strategies 20 student-teachers used in a CT performance assessment (PA). We implemented cognitive laboratories with think-alouds (Leighton, 2017). Each think-aloud was then converted into a color-coded flow-chart that visually displays the step-by-step process of each student as they read through the task instructions, the document library and wrote their answer (represented by colors). The protocol was:
• Segmented into “steps” as indicated by transitions (e.g., from reading one document to another).
• Color coded for act (e.g., reading document).
• Put into an Excel matrix with columns representing segmented units
• Time tagged and coded for the appearance of facets of CT.
• Represented as flow-charts—boxes and arrows indicating sequence (Fig. 1).
The flow-charts were interpreted as visual and quantitative characterizations of the students’ CT. In a companion study, the validity of flow chart interpretation was examined (Ronderos et al., this symposium).
4. Data Source and Analysis
The data sources include the 20 cognitive laboratories conducted in Switzerland (10) and Colombia (10). The analysis is described above.
5. Results
The outcome was the creation of a flow-chart that depicted the progression of student cognition throughout the PA task. The flow-chart effectively classified the cognitive processes involved and the corresponding time allocations. Furthermore, the associated study demonstrates that the flow charts provide an accurate portrayal of the cognitive process and are linked to desirable results.
6. Scholarly Significance
This work is a methodological contribution. Whereas cognitive processes have been captured in think aloud protocols, the thinking is typically characterized by static variables. Yet thinking is a process. This study provides one possible method that captures that process.

Authors

• Natalia Ronderos, University of Zurich

• Richard J. Shavelson, Stanford University

• Doreen Flick-Holtsch, University of Zurich

• Roza Nalbandyan, Stanford University