Challenges of International Contexts on Performance Assessment Translation and Adaptation

• In Event: Conceptual and Methodological Challenges of International Performance Assessment of Critical Thinking

Abstract

Objectives
We examine the types of adaptations needed for internationally comparable, valid, and fair PAs of university students’ CT. We ask: for a specific PA, “Migrants”, how much and what types of adaptations are needed for each culture, to maintain valid measurement of the construct?
Theoretical Framework
For universities that seek to benchmark themselves internationally the comparability of the scores of students taking different versions of a test is essential. Comparability is the extent to which: “… students’ scores can be validly compared … even if those scores come from measurements taken at different times, in different places, or using variations in assessment content and procedures. …” (Berman et al., 2020, p. 14). For comparability, task equivalence is necessary. For this study we focus on contextual and experiential equivalence. The challenge of producing comparable, valid, and fair assessments across cultures is daunting, especially for PAs due to the large document libraries, extended open-ended student responses, and scoring (Ronderos et al., 2021; Davey et al., 2015)
Method
To study comparability of PA translation/adaptation, we conducted case-studies in the U.S., Switzerland, Colombia, and Germany. Methodologically this study followed the Ronderos et al. (2021) framework for the adaptation of PAs. An international team representing the 4 countries met online periodically to reach agreement on the construct and facets to be measured, the content of the international PA, and the empirical work necessary to address the research question. For the development of the international version of the PA student data were gathered through cognitive laboratories.
Data Source and Analysis
The data sources for the adaption process were twofold: (i) notes or memos from meetings of the international team, and (ii) analysis of cognitive lab think alouds and interviews conducted by the four country teams.
Results
The different linguistic/cultural contexts posed substantial challenges to the development and adaptation of the basic version of the Migrants PA. One challenge is the tension between a country’s culture and the construct. In meetings, discussions repeatedly focused on balancing the desire to keep the international PA “close enough” to the students at the four sites, so that they engage in CT, but “far enough” so that their emotional engagement with the task did not hinder CT. Though the different stages in the process the tension between culture and construct, and between comparability and cultural sensitivity, was manifested. The discussions from the four sites, after gathering data from the cognitive laboratories, illustrated the same tension.
Scholarly Significance
This work contributes to enhancing the comparability, validity, and fairness of international comparisons of higher-order cognitive skills in higher education and beyond. It addresses the annual theme of the call, by “[interrogating] to ensure equitable processes and results”. (AERA, 2022, p.1)

Authors

• Natalia Ronderos, Institute for Education
  Presenting Author

• Doreen Flick-Holtsch, University of Zurich
  Non-Presenting Author

• Richard J. Shavelson, Stanford University
  Non-Presenting Author

• Henry I. Braun, Boston College
  Non-Presenting Author