Explaining Cross-National Noninvariance of Citizenship Values Concepts in the International Civic and Citizenship Education Study (ICCS) 2009

• In Event: A Diversified or Homogenized Landscape in Cross-Cultural Education Systems?

Abstract

The aim of this paper is to shed light on reasons for measurement non-invariance of two citizenship values concepts measured in the ICCS study. First, we aimed to establish whether concepts capturing student citizenship values measured for the ICCS are comparable (measurement invariant) across all countries participating in the study. Our preliminary results showed that the concepts investigated are not necessarily comparable across all countries involved in ICCS. We therefore intend to expand the purpose of our investigation and address in a second step the reasons for measurement non-invariance by testing weather contextual country-level variables could be explanatory for item bias.

In this article, we focus on two of the ICCS (which is currently the most comprehensive cross-national source of information regarding civic and citizenship education) measures (scales) frequently used for secondary data analysis purposes: conventional and social movement student citizenship values. We review the literature around these concepts and present concisely the notion of “measurement invariance”. We than present results of measurement invariance tests showing that the concepts are not necessarily comparable across all countries involved in ICCS and their use for secondary data analysis can be more or less feasible across all countries depending on the research questions and the methodology applied. In a last step we intend to detect items that display large cross-country differences and identify possible sources of bias in economic and cultural country characteristics (e.g. cultural and historical similarity/dissimilarity, experience with democracy and forms of political participation),


To check measurement invariance properties, we use a multi-group factor analytical framework (MG-CFA) and check for three levels of measurement invariance — configural, weak (metric) and strong (scalar). To detect the causes of non-invariance, we verify whether contextual country-level variables could be explanatory for item bias. To do so we employ multilevel structural equation modeling (MLSEM).

The data came from 37 countries in which the IEA-ICCS study 2009 was conducted in. We use the information from student responses (14-year-olds) on two sets of items measuring citizenship values.
The sample sizes vary between 1,964 in the Netherlands to 6,576 in Mexico with a total of 140,293 across all 37 countries.

Preliminary results show that the concepts are not comparable across all countries involved in the ICCS but could be used for cross-country comparisons for sub-groups of countries (e.g. Latin American countries). Results of further analyses will potentially show how we could explain item bias for the countries for which measurement invariance does not hold. Furthermore finding the source of bias can deliver useful information on how certain scales may be improved for cross-cultural research.

The study has scientific and practical implications. It illustrates a methodological approach useful for researchers in order to conduct comparisons of scale means across countries and interpret it in a meaningful way. It provides information on construct equivalence across countries for a wide scientific audience interested in performing secondary analysis of ICCS data as well as creates general awareness regarding the need of assessing properly the validity of questionnaire data in international surveys.

Authors

• Maria Magdalena Isac, European Comission, Joint Research Center

• Dorota Weziak-Bialowolska