Assessment of the Multivariate Outlier Approach for Differential Item Functioning: A Monte Carlo Simulation Study

• In Event: Addressing Practical Issues in Measurement Invariance Research: Methodological Investigations and Advances

Abstract

Objectives and Theoretical Framework
Differential Item Functioning (DIF) is used to detect item bias amongst groups. Prior research has primarily focused on the two groups (e.g. gender) case, though in practice variables with more than two groups (e.g. ethnicity) are frequently of interest. Although the multiple group case has not been heavily researched, several proposed methods do appear to be useful for DIF detection in such cases. These approaches include: Generalized Mantel-Haenszel test (GMH; Penfiled, 2001), Generalized Logistic Regression (GLR; Magis, Raîche, Béland, & Gérard, 2011), Lord’s Chi-Square test (Kim, Cohen, & Park, 1995), the alignment procedure (Asparouhov & Muthén, 2014), and the multivariate outlier approach (Magis & DeBoeck, 2011).
Each of these techniques addresses the problem of DIF detection in a different manner, but all use an estimate of the latent trait being measured, and this latent trait must not be influenced by DIF items if the method is to work correctly (French & Maller, 2007). One method that does not require this item purification is the multivariate outlier approach, which identifies an item as containing DIF when the value of a particular DIF statistic (e.g. Mantel-Haenszel log-odds ratio) for the item is an outlier in the multivariate distribution of the statistics across items. Preliminary results show that this approach has the potential advantage of better controlling the Type I error rate in the presence of multiple DIF items, without the need for item purification (Magis & DeBoek, 2012). Given the promising preliminary results, and the lack of research on this method, the focus of this study is to compare the multivariate outlier approach to other DIF detection methods for more than two groups.
Methods
This simulation study is designed to compare the multivariate outlier approach with several DIF detection methods for multiple groups, including: GMH, GLR, Lord’s Chi-square test, and the alignment procedure. In order to compare these methods, a variety of conditions were included in the simulation (see Table 1). These conditions focus on the number of groups, reference group sample size, group size ratio, method, purification, impact, DIF magnitude, and percent of items containing DIF.
Results and Scholarly Significance
Preliminary results for the two and three group case suggest that GMH has consistently higher power than other approaches, across sample sizes and DIF magnitudes when no contamination was present (Figure 1). GMH also had the lowest Type I error rate when contamination (i.e., item 2 did not have DIF) was present (Figure 2), and GMH had the highest power across the amount of DIF present (Figure 3). The multivariate outlier approach controlled the Type I error rate under most conditions, but also had lower power than GMH when DIF was present.
This study will add to the existing literature in DIF detection through a thorough examination of the multivariate outlier approach, which may be preferable in the presence of DIF contaminated latent trait estimates. In addition, this study extends prior work by examining the 6 groups case, and including the relatively new DIF alignment procedure.

Authors

• Julianne Edwards, Ball State University

• William Holmes Finch, Ball State University