University Students’ Critical Thinking – Evaluation of Performance and Thought Processes when Using Multiple Online Sources

• In Event: On the Comparability of Cross-National Assessment of Critical Thinking

Abstract

1. Objectives
When learning with freely accessible online resources, students are confronted with the challenge of encountering unreliable, inaccurate, or incorrect information(Osborne et al., 2022). To deal critically with sometimes contradictory sources, to evaluate them, and to come to a well-founded conclusion Critical Thinking(CT) is essential for academic success(Weber et al., 2019). Despite the relevance of promoting CT at universities, it is often limited or not explicitly anchored in the curriculum(Wineburg et al., 2018). A prerequisite for the effective promotion of CT during studies is the valid assessment of the skills, its underlying facets, and thought processes, whereby the specific complexity of the development of a CT assessment must be considered in an international comparison(Berman et al., 2020).
2. Method/Data
To validly measure students' CT skills and gain insights into performance and thought processes, an international consortium(Germany/Colombia/Switzerland/USA) developed a holistic scenario-based performance assessment(PA) with realistic simulation of a decision-making situation(Braun et al., 2020). The PA contains pre-selected (ir)relevant, (in)reliable, and sometimes contradictory sources/information that are/is available to students when responding to the task and making decisions. Cognitive Labs(CogLab) were conducted following common protocol. Data were collected through students' responses to the PA as CT-performance and response processes through Think Alouds(TA)(Leighton, 2017). This presentation focuses on the research question of how CT is reflected in students' performance(result-oriented) compared to the thought processes involved in solving the PA(process-oriented)(e.g. regarding the selection of sources, considering relevance and credibility).
After a validation-study, the CogLab-study comprised surveys at the four locations in the winter term of 2023/2024, each with N=10 students(Table1), at the end of their bachelor’s or the beginning of their master's degree. To determine CT-performance, the written responses to the PA were evaluated by two trained raters(with strong ICC;Mean=.75) using a harmonized criteria-based scoring scheme with a 5-point Likert scale on four theoretically-defined CT-facets(analyzing/evaluating information; recognizing/weighing consequences; formulating/communicating coherent arguments; perspective taking(Braun et al., 2020) with 13 sub-dimensions. To gain insights into the thought processes, the transcribed TA from the CogLabs were analyzed qualitatively.
3. Results
On average, students completed the PA in 114.28 minutes, wrote 555.40 words in their responses to the PA, and the transcribed TA comprised 8555.30 words. The results of the CT-performance(Table2) from the CogLabs indicate insufficient CT-skills of the students(Mean=2.827,SD=.773) with a strong variance in all CT-facets, especially in ‘analyzing and evaluating the information with the number of sources used(Mean=3.750,SD=.920) and the correct evaluation of the quality of information(Mean=3.750,SD=.950). The initial analyses of transcribed TAs show CT-performance- and process-oriented differences regarding the use of sources, including the number of sources used in performance(Mean=4.500,SD=.970) and processes(Mean=8.000,SD=.000).
4. Significance
These results are consistent with previous research indicating deficient CT-skills(Osborne et al., 2022). The results of the qualitative content analyses of the thought processes during PA processing provide additional important indications of which factors influence students' behavior when processing information. The findings can contribute to the further development of research on valid CT measurement in international studies(Padilla & Leighton, 2017) and provide important hints for the promotion of CT at universities.

Authors

• Katharina Depré, Johannes Gutenberg University of Mainz

• Olga Zlatkin Troitschanskaia, Johannes Gutenberg University of Mainz

• Dominik Braunheim, Johannes Gutenberg University of Mainz

• Richard J. Shavelson, Stanford University

• Paul Frederik Hodes, Johannes Gutenberg University of Mainz