The development and validation of a multi-dimensional cosmopolitan worldview scale: an IRT approach

• In Event: Roundtable Session (Tuesday V)
  In Refereed Round-Table Session: Reflections on cosmopolitanism and education: Developing cosmopolitan sensibilities for a sustainable future

Proposal

As practitioners and scholars expand the literature on cosmopolitanism and develop best practices in educational programs that cultivate the cosmopolitan ideal, reliable and valid measures for cosmopolitan worldview are needed to benchmark cosmopolitan education outcomes, evaluate program effectiveness, and understand student’s learning progression. Deardorff (2011) emphasized that assessing intercultural competence requires a clear defined model for intercultural competence, a set of prioritized goals and measurable objectives, and a series of assessment approaches, methods and tools to collect direct and indirect evidences. Measuring cosmopolitan dispositions should also follow the three principles Deardorff outlines. Rather than simply documenting the process and implementation of cosmopolitanism education programs, gathering evidence for the development of the cosmopolitan worldview is essential to the assessment process. There is a large body of evolving literature on the theoretical discourse on cosmopolitanism in sociology that other domains draw upon. However, most research in cosmopolitanism in sociology as well as many other domains remains conceptual and qualitative.

Several scales measuring constructs related to cosmopolitanism, or scales measuring one of the cosmopolitanism dispositions have been developed by researchers in various fields including environmental sciences, economics, and psychology (Morais & Ogden, 2011; Kelley & Meyers, 1992; Hunter et al., 2006; Leung, Koh & Tam, 2015; Cleveland, Laroche & Papadopoulos, 2007 & 2009). However, all of these scales utilize Classical Testing Theory as the validation method, and some have unsatisfactory results in internal structure, content validity, or reliability. The most rigorously designed and robustly validated scale by Cleveland et al. (2007) is intended to be used outside of an educational setting.

Using Wilson's (2004) four building blocks approach to instrument construction as methodological framework, this paper defines the multi-dimensional construct map of cosmopolitan worldview, documents the rigorous process of items design, explains the construction of the outcome space, and reports the fitting results of several measurement models. The quantitative analysis in this paper relay on Item Response Theory (IRT), which is a model based test theory that estimates both the person’s ability and the item difficulty in its application. By applying IRT analysis, the author was able to obtain a standard error of measurement that differs across scores, equating different test forms, obtaining unbiased estimates of item properties from unrepresentative samples, and achieve interval scale properties by justify measurement models instead of score distributions (Embretson, 1996). Validity evidence based on construct, response processes, internal structure, relation to other variables, consequences of using and fairness, as well as two forms of reliability evidence are presented in the paper.

In this presentation, I will emphasize on the rationale behind the multi-dimensional construct map of the cosmopolitan worldview, highlight the items design, and some future steps for the next phase of this study. Recommendations on how to use the scale and interpret the results will be made while introducing the IRT model fit analysis. A very brief overview of the other validity and reliability findings will also be presented.

Author

• Yidan Zhang, University of California, Berkeley