How Critically Do Students Engage with AI?

• In Event: Innovation in Cross-National Critical Thinking Assessment: AI Integration, and Innovations in Performance-Based Measurement

Abstract

1. Objectives
This paper aims to advance knowledge on the formative use of performance assessments (PAs) to support the development of critical thinking (CT). Specifically, it explores how students critically engage with generative AI (ChatGTP) while solving, revising and developing CT performance tasks.
2. Theoretical Framework
The study draws on the iPAL framework for performance assessments of CT (Braun et al., 2020), explicitly incorporating metacognition into the construct definition as outlined by Facione et al. (1990). It also builds on prior research on the formative use of CT performance tasks to foster students’ CT (Cargas et al., 2017; Ronderos et al., 2025).
Regarding the intersection of AI and CT, the study is informed by emerging literature on students' critical engagement with generative AI tools in educational contexts, which has seen in this technology opportunities for learning (Federiakin et al., 2024) and for enhanced CT (Gonsalves, 2024; Walter, 2024). Additionally, CT has been identified as a crucial facet for effective prompt engineering (Federiakin et al., 2024). However, there are also concerns that AI can promote "unlearning” (Abbas et al., 2024 in Federiakin et al., 2024) and result in dependency and cognitive off-loading (Lodge et al., 2023; Ratten & Jones, 2023 as cited in Gonsalves, 2024). The difference between AI resulting in improved CT or weakened cognitive ability is dependent on the critical engagement of the student (Gonsalves, 2024).
3. Methods
The study will be implemented within the context of a master's-level module at the University of Zurich in Fall 2025, with approximately 25 students.
● To investigate students' critical use of AI, participants will have open access to ChatGPT while they (1) complete different CT performance task, (2) revise and improve their answers, and (3) develop a PA of CT.
● Students will submit all scripts of their ChatGTP interactions, and a sample will be analyzed using the CT facets from Braun et al. (2020) and the metacognitive dimensions from Facione et al. (1990).

4. Data Sources
Data sources include:
● Students’ scripts with ChatGTP, generated while solving the PAs of CT, revising them, and developing their own.
● Student survey responses regarding their perceptions of their critical use of AI, frequency of use, and other control variables.
● Students’ PA responses to other PA tasks of CT.

5. Anticipated Results
We expect that students with higher levels of CT, as seen in their performance in previous PAs of CT, will have more critical engagement both in terms of: (1) prompt engineering, and (2) evaluating the response given by Chat GPT. We also expect that students’ critical engagement will occur mainly while revising their answers and developing their own PA of CT, and to a lesser extent while they solve the PA.

6. Scholarly Significance
This study contributes to ongoing discussions on the operationalization of CT, the design of PA, and their pedagogical use, as well as on the critical use of AI tools in higher education.

Authors

• Natalia Ronderos, University of Zurich

• Doreen Flick-Holtsch, University of Zurich

• Julian Patricio Mariño von Hildebrand, Universidad de los Andes

• Maren Oepke, University of Zurich