Profile Analysis: A Generalization of Differential Item Functioning Analysis

• In Event: Roundtable Session 117
  In Roundtable Session: The Whole Is Greater Than the Sum of Its Parts: Research on Aggregated Test Scores

Abstract

Sometimes the paradoxical situation in measurement arises, where it is required to report on a single (unidimensional) scale while on the other hand the framework that forms the basis for the item construction is highly multidimensional. A good example is the international Survey PISA where a moderate to high number of supra-individual entities, the participating countries, are compared and ranked with respect to different area’s of competence: Reading, Mathematics and Science literacy. At the same time, the frameworks prescribe that these three areas should be described by a complex model of different external (e.g., type of stimulus material, item format) and internal (e.g., inferred processes) criteria. This situation entails two important questions: (i) Is it reasonable to describe the performance on a highly heterogeneous set of items by a simple unidimensional measurement model and (ii) if the answer to the first question is affirmative for a single country, how can one assess the invariance of the scale (construct) across countries.
Profile Analysis (Verhelst, 2011) is a method to address the latter question: instead of searching for differential functioning of items (across countries, in the example), a method is proposed to search for different functioning of subsets (or categories) of items. It is argued that the method explains variance in the data beyond (and orthogonally to) the variance explained by the measurement model, and consequently, that the new methodology offers a set of complementary analyses to the results offered by using a relatively simple unidimensional measurement model.

Author

• Norman D. Verhelst, Eurometrics