Comparability of Critical Thinking Performance Assessment Across Colombia and Switzerland

• In Event: Conceptual and Methodological Challenges in Comparability of International Performance Assessment of Critical Thinking

Abstract

1. Objectives or Purposes
We examine the comparability of a specific Performance Assessment, “Migrants”, across four countries. We ask, for two of the four sites (Switzerland and Colombia): (1) Does the Performance Assessment Migrants measure the same construct (or subset of construct facets) across countries? (2) Is the proposed Critical Thinking interpretation of cross-national student performance supported empirically with cognitive process and self-report evidence?
2. Theoretical Framework
For universities that seek to benchmark themselves internationally the comparability of the scores across countries, and different language versions of the test, is essential. The challenge of producing comparable, valid, and fair assessments across cultures is daunting, especially for Performance Assessments due to the large document libraries, extended open-ended student responses, and scoring (Ronderos et al., 2021; Davey et al., 2015). Comparability is the extent to which: “… students’ scores can be validly compared … even if those scores come from measurements taken at different times, in different places, or using variations in assessment content and procedures. …” (Berman et al., 2020, p. 14). For comparability, equivalence is necessary. This study focuses on response process equivalence and contextual and experiential equivalence.
3. Method
To study the comparability of Performance Assessment translation/adaptation, we conducted case studies in the U.S.A, Switzerland, Colombia, and Germany. Methodologically this study followed the formal adaptation alternative to Performance Assessments of Critical Thinking, presented in Ronderos et al. (2021). An international team representing the four countries collaboratively developed the international version of the Performance Assessment, incorporating feedback from students at the four sites in the development process. The Performance Assessment was then translated and adapted for the four sites and each site implemented cognitive laboratories with the think-aloud process (Leighton, 2017). The think-aloud and follow-up interview were then transcribed, coded and analyzed, using the Critical Thinking assessment framework (Braun et al., 2020) as cognitive model for coding, as well as looking at experiential and contextual equivalence (Ursin et al., 2022; Solano-Flores, 2011).
4. Data Source and Analysis
The data sources are the 20 cognitive laboratories implemented in Switzerland and Colombia.
5. Results
Overall, we find equivalent response processes between countries. We saw, in both countries, occurrences of all of the Critical Thinking facets defined in our Critical Thinking model (Braun et al., 2019). There was variation in the frequency of occurrences, both within and between countries.

6. Scholarly Significance
This work contributes to enhancing the comparability, validity, and fairness of international comparisons of higher-order cognitive skills in higher education and beyond.

Authors

• Natalia Ronderos, University of Zurich

• Doreen Flick-Holtsch, University of Zurich

• Julian Patricio Mariño von Hildebrand, Universidad de los Andes

• Richard J. Shavelson, Stanford University

• Henry I. Braun, Boston College

• Maria Camila Perfetti, Universidad de los Andes