The Potential of Anchoring Vignettes to Increase Intercultural Comparability

• In Event: How Can Noncognitive Skills Be Fairly and Comparably Assessed Across Different Groups?

Abstract

International studies face the problem of cultural-specific answering biases which makes it difficult to compare research results across different countries or cultures. In the past, anchoring vignettes have been used to reduce these kind of answering biases (e.g., King, Murray, Salomon, & Tandon, 2004). Anchoring vignettes are a technique to account for a person’s use of the full range of the answering scale during data post processing. As a first step, culture-bound answering tendencies are captured by asking participants to assess a person on a specific characteristic (e.g., achieving motivation). This person is either described as having a low, medium or high level of this characteristic (anchoring vignette). Afterwards, participants give self-assessments on the same characteristic. During post assessment data processing, self-assessments are adjusted or anchored in accordance with their culture-bound answering tendency.
The aim of the present analysis is to broaden the research literature on the scope of anchoring vignettes’ capacity to improve data quality. Particularly, the study focuses on the question whether anchoring of self-assessments leads to internationally more reliable and valid data in comparison to non-anchored self-assessments. The study applies different criteria for reliability and validity, such as factorial structure. The used data set contains 15-year old students in more than 60 different countries worldwide. Moreover, data on three different types of anchoring vignettes were included in the study. The three vignettes describe achieving motivation, organization, and test anxiety. Vignettes were used to anchor the respective constructs and constructs that are only vaguely related (other motivational or personality factors).
First analysis show inconclusive results across the different anchoring vignettes and scales. Results of exploratory and confirmatory factor analyses indicate that the data quality for some of the constructs can in fact be improved by anchoring, while for other self-assessments, anchoring is less successful than hoped for. The results will be discussed with regard to possible consequences for research methodology and reporting.

Authors

• Tamara Marksteiner, University of Mannheim

• Susanne Kuger, German Institute for International Educational Research

• Eckhard J. Klieme, German Institute for International Educational Research