Differential Item Functioning Analysis in International Large-Scale Assessments: From a Traditional to a Network Perspective

• In Event: Future Directions of Educational Measurement in International Large-Scale Assessments

Abstract

In this presentation, we formulate how differential item functioning (DIF) can be analyzed from a network perspective in the context of international large-scale assessments, in contrast to the traditional methods based on item response theory (IRT) modelling.

DIF implies problems of measurement invariance in a test as it indicates an interaction between item responses and background characteristics (e.g. country or gender). In international large-scale assessments, DIF has been typically investigated at the country-level via IRT modelling (see, e.g., von Davier & Sinharay, 2014). Network psychometrics has focused on studying psychological and educational phenomena as dynamic systems (for a wide discussion see, e.g., Marsman et al., 2018). A network is used to graphically represent the system: item responses are represented as nodes and links correspond to interactions between them. Under a network perspective, phenomena are understood as processes that emerge from the interactions between variables. For instance, a network of cognitive items in which all the items are mutually connected implies that a single cognitive process (i.e. a single dimension) emerges from the assessment. Recently, a latent variable representation has been proposed to investigate the structure of interactions in a network. The main idea is that the connectivity matrix, which corresponds to the interactions between every pair of nodes, may be approximated by a low-rank matrix that lets us investigate the number of dimensions and the structure of interactions involved in the network (Marsman et al., 2018; Marsman, Maris, Bechger & Glas, 2015).

We illustrate how DIF analysis can be performed under a network perspective using PISA (Programme for International Student Assessment) 2012 data. This framework lets us investigate interactions between item responses and individual background variables (e.g. gender or socio-economic status), but higher-order interactions related to the country-level are also analyzed.

The key ingredient of our proposal is that it analyzes the joint distribution of item responses and background characteristics. We mainly focus on the latent variable representation of the network to investigate DIF as a dimensionality problem that involves item and background responses. This perspective helps to understand the relations between them and distinguish between construct related and construct unrelated DIF. Our proposal extends DIF analysis in international large-scale assessments to more than one variable and lets us investigate mutual interactions between background characteristics and item responses. In addition, our proposal contributes to investigating measurement invariance in networks psychometrics.

This presentation focuses on the methodological proposal to perform DIF analysis in international large-scale assessments from a network perspective. We argue that a network perspective is convenient to investigate DIF in educational surveys and we formulate a distinction between construct-related and construct-unrelated DIF.

Authors

• Edwin Cuellar Caicedo, Cito

• Timo Bechger, ACT, Inc.

• Maarten Marsman, University of Amsterdam

• Gunter Maris, ACTNext