An Investigation of the Efficacy of DIF (Differential Item Functioning) Refinement Procedures

• In Event: Explorations in Differential Item Functioning

Abstract

We conducted a simulation study of criterion refinement in Mantel-Haenszel DIF analysis, focusing on the relevance of DIF pattern: unbalanced (all in one direction) or balanced. We examined Type I and II error rates of the ETS DIF flagging rules, as well as properties of the DIF estimates. Our main findings: In small samples, refined and unrefined methods behaved similarly because refinement led to few exclusions from the criterion. Differences between the methods were also small in long tests. When the total sample size was 1000 and the test was 20 items long, refined analyses performed better than unrefined with unbalanced DIF. Unrefined analyses performed better with balanced DIF, where refinement sometimes produced bias in a formerly balanced matching variable.

Authors

• Rebecca Zwick, Educational Testing Service

• Lei Ye, Educational Testing Service

• Steven Isham, ETS