Small Decision, Big Consequence: How Australia Moved to the "Good Quadrant" in PISA (Programme for International Student Assessment)

• In Event: Roundtable Session 46
  In Roundtable Session: Methodological Choices and Their Consequences in Education Research

Abstract

Objective
This paper describes how a methodological decision to make a small change in the way ‘equity’ is measured in PISA resulted in Australia moving from ‘high quality, low equity’ in PISA 2000 to ‘high quality, high equity’ in PISA 2003. The objective is to highlight some of the less discussed processes by which methodological decisions are made in research, and to demonstrate that they can have far-reaching consequences.
Theoretical Framework
This study is located within the sociology of measurement and numbers (Author XXXX), an emerging field of research that uses resources from Science and Technology Studies (STS) and actor-network theory. Sociologists of numbers are interested in following numbers-in-the-making and exploring how numbers are developed and how they come to be stabilized as believable and useful understandings of the world. They are also interested in the circulation of these numbers in social worlds.
Methods
This study is based on in-depth interviews with PISA experts who were well placed to discuss the debates around the choice of indicators to measure equity in PISA.
Data Sources
In addition to interview data, this study also used PISA reports, including its technical reports and other OECD publications, as well as published articles which debated how to measure equity.
Results
This research found that in the case of Australia, a small change in the number of indicators used and the weight attached to the indicators moved Australia from ‘high quality, low equity’ in PISA 2000 to ‘high quality, high equity’ in PISA 2003 without any significant change in Australia’s school system itself – the move to the ‘good quadrant’ was the consequence of a methodological choice in measuring equity.
The study demonstrated that rendering abstract notions such as ‘equity’ measurable and comparable depends on the attribution of measurable characteristics to the phenomenon. Decisions about which indicators to use, and how to weight them, are made through ‘field testing’ and through negotiation within a community of experts. Whilst not ‘arbitrary’, such practices are neither ‘objective’ nor self-evident. Rather, they are negotiated and ‘interested’ decisions.
Scientific or scholarly significance
This study substantiates the argument that all research findings are theoretical, methodological and analytical artifacts – an understanding that is often under emphasized in research. Education research – especially research explicitly produced for policy – is under pressure to present unambiguous results and definitive solutions for policy problems – and this encourages certainty and strong claims. This study makes a case for education researchers to employ multiple theoretical and methodological approaches, engage in inter-disciplinary research, and maintain modesty in research claims.
By focusing on the processes of measurement and comparison, this paper highlights three issues: the contingent and uncertain nature of research knowledge (Barad, 2007); the systematic ways in which such uncertainty and contingency disappear in research processes (Latour, 1987; Knorr-Cetina, 2007); and the significant consequences of methodological choices. Given the influence of PISA, this call for reflexivity in research and modesty in research claims is timely and apposite.

Author

• Radhika Gorur, Deakin University