Some Drivers of Test Item Difficulty in Mathematics: An Analysis of the Competency Rubric

• In Event: Exploring Reading and Mathematics Item Difficulty: Teaching and Learning Implications of PISA (Programme for International Student Assessment) Survey Data

Abstract

This paper is concerned with the empirical validation of the competency rubric described in the previous paper. Using items developed for the PISA 2012 survey, and data collected as part of an extensive field trial of the items conducted during 2011 in some 67 countries, we use multifaceted Rasch modelling and latent regression to examine the following three questions:
1. What is the level of agreement among raters when they apply the competency rubric?
2. Does each of the competencies capture different dimensions of cognitive complexity in the items?
3. To what extent do ratings of the cognitive complexity account for (predict) the difficulty of the tasks for students?
The data available for this analysis included the ratings of over 150 items by eight mathematics educators. The raters had been previously trained in the use of the rubric using a different set of items. Student response data were available for over 10,000 students for each item. The outcomes of the validation indicate very strong inter-rater agreement, clear discriminant validity and they suggest that the six competencies can together account for up to 70% of variance in item difficulty.
This research has the potential to significantly advance our understanding of the central drivers of item difficulty in mathematics. It carries implications for understanding and describing growth in mathematical competence, and therefore for the place of the competencies in the mathematics curriculum and in the daily work of mathematics teachers. These implications are likely to be relevant far more widely than in the PISA context alone.

Author

• Raymond J. Adams, Australian Council for Educational Research