Knowledge by the Number: A Systematic Research Review on International Comparisons of School Performances

• In Event: Numbers, Education, and the Making of Society: International Assessments and Their Expertise

Abstract

1. Purpose and objectives: International comparisons of education performances (ICEP) are of vital concern in current research and policy discourses (Carvalho, 2009) and is important to analyze. Thus, we did a systematic research review concerning large scale assessments such as the OECD and the PISA (Program for International Study Assessments) and the IEA, with TIMSS (Trends in International Mathematics and Science Study) and CIVED (Civic Education Stud)y. What characterizes this field of study and its knowledge contributions?
2. Perspectives: We are designing our study as a systematic research review with a combination of aggregative and configurative approaches (Gough, 2012) in a transparent way. However, the focus is to capture the research field and its knowledge contributions and not specific interventions and their outcomes. Given this we focus on the content of research in terms of arguments and practical reason (von Wright, 1983) in combination with styles of reasoning (Hacking, 1992) in this research.
3. Methods: A design for systematic research reviews was used. The search engine “Discovery” was used to identify ICEP publications. Specific criteria for filtering publications with focus on empirical comparative studies in peer reviewed journals was used. Our focus is the PISA and TIMSS and CIVED studies.
4. Data sources: The achieved data in terms of research publications were filtered as is shown in table 1. Of the identified 8 744 publications 4 786 were published in peer reviewed journals. The filtration process ended up with 135 articles which fit our selection criteria.
5. Results: Four main results are highlighted here: (a) A small share of the publications presented empirical studies of international comparisons. Many publications were second order comments, e.g. for policy and practice. (b) ICEP research mostly analyzed individual student performances and only used data from a specific research program. Few studies tested specific hypotheses. Most emphasized an interest in political and professional rather than an academic relevance. (table 2). (c) The publications were less about efficiency and rankings and more about disadvantages for different kinds of people. And (d) The research mostly developed statistical styles of reasoning (see e.g. Hacking, 1992) concerning what statements could or could not be put forwards (with quite a few exceptions).
6. Two main conclusions: We identified a specific and homogeneous field of knowledge objects (Kuhn, 1971) in the way these are formulated and how research inquiries are carried out. However, considering study objects (Lundgren, 2000) ICEP research is heterogeneous – with a large share of publications devoted to political and administrative issues. By means of the systematic research review we could identify a science-society agora where other actors – outside academic criticism- presented themselves in the field of educational policy discourses – translating e.g. PISA-results in strategies for system improvement or into policy advices.
6. Significance: Our results and conclusions are important in order to capture current strands in educational research and the working of science-society agora in education as well as implications of understanding education by thin descriptions (Porter, 2012) and looping (Hacking, 1995).

Authors

• Sverker S:son Lindblad, University of Gothenburg

• Daniel Pettersson, Gävle University College