Extending the Range of Item Difficulty in Reading for PISA (Programme for International Student Assessment) 2009

• In Event: Exploring Reading and Mathematics Item Difficulty: Teaching and Learning Implications of PISA (Programme for International Student Assessment) Survey Data

Abstract

In terms of improving learning, a major benefit of developing an item difficulty framework for an assessment is to facilitate reporting to stakeholders that describes increasing proficiency in systematic and interpretable terms. A further benefit, from a technical perspective, is to guide the construction of items in the required range of difficulty for a given assessment, so that it will appropriately target the whole proficiency range of the population. The second of these purposes will be the focus of this paper.
In PISA 2000, the first administration of PISA, reading was reported against five described levels of increasing proficiency. While the measurement of student achievement in reading was accurate across the range of the scale, little qualitative information was available about what students at the extremes of the scale knew and could do. Across the 28 OECD countries that participated in PISA 2000, a substantial proportion of 15-year-olds – 6% on average – performed below the lowest described level; there was only one item in the administered pool that was located below Level 1, so it was not possible to generate any broad description of what students performing in this low range could typically do as readers. A refined description of the reading capacities of extremely high-performing students was also unavailable, for similar reasons. Accordingly, for PISA 2009, an attempt was made to develop sufficient items to populate the regions of the scale above Level 5 and below Level 1, so an additional level of proficiency could be described at each end.
An iterative process was adopted, involving the simultaneous refinement of an item difficulty framework that was drafted during the PISA 2000 cycle (OECD, 2002), and the development of the items themselves. For the latter, teams of professional test developers in Australia, Belgium, Germany, Japan and Norway collaboratively undertook item development in 2006-2007, using material contributed both by the test development centres and by national centres in 30 of the countries that were participating in the 2009 survey.
In a field trial conducted in 2008, the items were administered to 69,570 students in 67 countries, in 46 languages. The field trial yielded 17 very easy items (from an administered pool of 188 new items), of which nine were selected for the main survey. These items were calibrated below the previously defined Level 1. A more modest success was achieved in applying the item difficulty framework to build very hard items. Nevertheless, it was possible to develop a described Level 6 as well as a Level 1b (below the previous Level 1) for reporting PISA 2009 reading results (OECD, 2010).
The paper will describe the process of developing and analysing very easy and very difficult reading items for PISA 2009, based on a 10-criterion item difficulty framework. The paper will discuss the results and potential of the work, and examine some of the associated technical and operational issues.

Author

• Dara Ramalingam, Australian Council for Educational Research