University Students’ Source Evaluation and Argumentation: Comparing Performance and Processes through Essays and Think-Alouds

• In Event: Innovation in Cross-National Critical Thinking Assessment: AI Integration, and Innovations in Performance-Based Measurement

Abstract

1. Objectives
As university students increasingly rely on freely accessible online resources (Osborne et al., 2022), the ability of Critical Thinking (CT), including the evaluation of information quality, development of evidence-based arguments, and formulation of well-reasoned conclusions, is essential for academic success (Weber et al., 2019). A key challenge in fostering CT is its valid assessment, which must capture both the product and the process of students’ reasoning. This study investigates how CT is reflected in students’ written performance (essays) and cognitive processes (captured through Think-Aloud transcripts) when responding to a scenario-based performance assessment (PA) task.
2. Theoretical Framework
An international research consortium developed a four-dimensional CT framework: (1) analyzing/evaluating sources; (2) formulating/communicating well-founded arguments; (3) recognizing/considering consequences; (4) writing effectiveness (Braun et al., 2020). Based on this, a scenario-based PA was designed to simulate a realistic decision-making situation, using pre-selected (ir)relevant, (un)reliable, and contradictory sources with low-/high-quality arguments on a controversial topic (regulation of migration). The task required students to write an argumentative essay integrating information from these sources and making a justified recommendation. CT performance was evaluated based on essay quality, while CT processes were examined through Think-Aloud transcripts. This presentation focuses on two CT dimensions - source evaluation and argumentation - and explores how students’ CT processes correspond with their written performance.
3. Methods and Data
Cognitive Labs were conducted in four countries in winter 2023/24, using a standardized international protocol. Data sources included students' essays as CT performance and Think-Alouds protocols during task completion to capture CT processes; 10 students per site participated (s. Table 1), all at the end of their bachelor’s or beginning of master's degree.
Essays were scored by two trained raters using a harmonized, criteria-based rubric with strong interrater reliability (ICC=.75). Think-Aloud transcripts were qualitatively segmented and coded (Mayring, 2015). Essay scores and Think-Aloud coded segment frequencies were descriptively and correlatively compared to examine the relations between students’ CT performance and processes.
4. Results
Students spent on average 114.28 minutes completing the PA task. Essays averaged 555.40 words; TA transcripts averaged 8555.30 words.
Essay scores showed variation in both target CT dimensions: “analyzing/evaluating sources” (M=3.750, SD=.898); “formulating/communicating arguments” (M=2.717, SD=.705) (s. Table 2).
Think-Aloud analysis revealed 287 segments related to source evaluation and 189 to argumentation (s. Table 3).
The number of references to source relevance/evaluation (31 segments) in Think-Alouds correlated negatively with essay scores in source evaluation (Pearson Corr.=-.903, Sig.=.000), indicating that frequent, superficial mention of relevance may not align with deeper, written source evaluation. Positive correlations emerged between integrative argument use in Think-Alouds (46 segments) and performance scores in this dimension (Pearson Corr.=.722, Sig.=.005) (s. Table 4).
5. Significance
These results confirm previous findings on the variability in students’ CT skills (Osborne et al., 2022) and underline the value of examining both cognitive processes and written performance. The joint analysis of essays and Think-Alouds provides deeper insights into how students evaluate sources and construct arguments, offering valuable implications for improving CT assessment and instruction in international higher education contexts.

Authors

• Katharina Depré, Johannes Gutenberg University of Mainz

• Olga Zlatkin Troitschanskaia, Johannes Gutenberg University of Mainz

• Dominik Braunheim, Johannes Gutenberg University of Mainz

• Thiemo Hagen, Johannes Gutenberg University of Mainz

• Richard J. Shavelson, Stanford University