Multilevel Item Response Theory in the Explanation of Cross-National and Cross-Linguistic Differential Item Functioning in Reading Comprehension 2009

• In Event: Perspectives on PISA (Programme for International Student Assessment) Reading

Abstract

A fundamental obstacle to overcome in the development of valid cross-national and cross-linguistic tests is ensuring that the items from which the test is composed measure the same construct in all national and linguistic contexts. The OECD’s Programme for International Student Assessment (PISA), which is administered in numerous countries and languages and assesses 15-year-olds in mathematics, science and reading, faces this problem of test equivalence.
Thorndike (1973) suggested that achieving pan-linguistic and pan-national equivalence amongst reading comprehension items is likely more difficult than with items measuring mathematics or science ability as reading is more intimately linked with localised language and culture. However, papers by Grisay, Gonzalez and Monseur (2009) and Grisay and Monseur (2007) report 82% common variance across 47 PISA 2000 test adaptations, demonstrating that cross-national cross-linguistic comparison in reading comprehension, in the PISA context, is possible, but not perfect.
Attempts have been made to partition the remaining variance, using components of variance analysis, to enable the determination of the sources of the differential item functioning (DIF) across countries and languages. They indicated that the largest proportion of the remaining variance was present in the interaction between the items and the country whereas the second largest proportion of the variance was between the items and the language.
Following a suggestion by Grisay, Gonzalez and Monseur (2009) a multilevel-item-feature (IFM) paradigm was suggested as a useful future path for research into cross-national and cross-linguistic DIF. The model has already been successfully applied, by Park and Bolt (2008), to illustrate and diagnose the cross-national achievement in specific mathematical skill areas in the Trends in International Mathematics and Science (TIMMS).
This study attempts to explain DIF in around 50 PISA 2009 Reading items, with regards to salient features of the items, as described in the PISA reading framework. It will achieve this by the application of a multilevel IRT paradigm which will allow the nesting of linguistic and national groups. The size of the effects of the salient characteristics of the items can then be gauged at both the level of the country and the level of the language. This will be the first study, to the best of my knowledge, to apply the multilevel item feature model to PISA reading items
The salient characteristics of the items to be investigated are the “text format” (continuous, mixed, or non-continuous), the “item format” (multiple choice, open constructed response, etc.), and the “aspect” (access and retrieve, interpret and integrate or reflect and evaluate).
The model will be both explanatory and diagnostic in nature; it will diagnose the ability of students across countries and language groups to deal with different text-formats, types and aspects. The results from the analysis of text-format will indicate which learners in which countries and linguistic groups have greater facility or difficulty in comprehending continuous, mixed and non-continuous texts. The analysis of the effects of item format will highlight specific countries’ performance with different item types. The exploration of aspect will highlight strengths and weaknesses in different levels of text-reader interaction.

Author

• Gareth McCray, Lancaster University