How to do things with ILSAs? International organizations and grade repetition in Latin America

• In Event: Highlighted session: Critical reflections on the development, use and policy impact of international large-scale assessments in education

Proposal

In the debate around the influence of international organizations (IOs) on national education policies, a key question is: what are the mechanisms through which this influence operates? This paper focuses on one possible mechanism, namely International Large-Scale Assessments (ILSAs). The issue of performative statistics is linked to the work of Desrosières (2015) and his analysis of official statistics, which paraphrased Austin’s classic How to do things with words? transforming it into How to do things with numbers? This article asks the question: How to do things with ILSAs? Can international assessments, because of their design, rather than the alleged “reality” of education systems, provide incentives for countries to adopt certain education policies?

IOs are not created equal; therefore, neither are ILSAs. IOs like UNESCO emphasize the human rights-based approach to education (Jones 1982; 1990). Meanwhile, others are heavily influenced by human capital theory, e.g. the World Bank (Jones 2007; Menashy 2013) and OECD (Bieber 2016). Consequently, there are nuances in their discourse around certain education policies: grade retention, for instance, has historically been regarded as an indicator of education systems’ “internal efficiency” (or lack thereof), and as such has often been harshly criticized by the human capital theory camp.

By applying a narrative policy framework (Schlaufer 2018) this article will show that policy debates have often cast grade retention as the villain. Over the past four decades it has been successively accused of four crimes: inefficiency (Psacharopoulos & Nguyen 1987); ineffectiveness (Jimerson et al. 1997; Jimerson 2001); inequity (Lorence & Dworkin 2006); and finally, insensitivity, due to the policy’s alleged pernicious effects on repeaters’ socio-emotional development (Demanet & Van Houtte 2016).

At the same time, this article argues that educational doctrines influence assessment design: Does the IO behind an ILSA works mostly from a human rights based approach or a human capital theory perspective? While the former tend to support grade-based assessments (e.g. TIMSS, PIRLS, the regional assessments), the latter tend to support age-based assessments (e.g. PISA). This variation in design may lead to substantial differences in these assessments’ results. Finally, these diverging results may in turn affect the conclusions drawn from the data with regards to the effectiveness (or lack thereof) of certain policies, such as, in this case, grade retention.

ILSAs are often cited in educational debates at the country level and have led to important changes in policy (Rutkowski 2016). They are prioritized as sources of information about policy effectiveness because of their alleged independence and perceived rigor, particularly as compared to national sources in developing countries: ILSAs are seen as politically above the fray and methodologically beyond reproach.

However, “there are no specific research questions posed by TIMSS or PISA study centers” (Rutkowski 2016, 25) or by other ILSAs, none of which is designed to evaluate the effectiveness of a particular policy. Because ILSAs are not meant to answer a specific research question, their design is not optimized with a view to this purpose. In other words, although ILSAs may collect information about a broad range of inputs, processes, outputs and outcomes, their design does not allow for the isolation of any specific causal relationships between, say, a given input, and a given outcome. One can think of ILSAs as a sort of omnibus study that, despite often being used to answer every possible question involving educational outcomes and predictors, is not specifically designed to answer any of them.

Consequently, their results must be examined very cautiously before drawing causal inferences related to policy (Rutkowski & Delandshere 2016). If the characteristics of an assessment’s design are not considered, findings from ILSAs may mislead policymakers. They may conclude –based on data from an ILSA– that a policy is either helpful or harmful for, say, academic achievement, when in fact that apparent effect may stem from that ILSA’s own design.

Geographical scope

The analysis focuses on four Latin American countries (namely Costa Rica, Dominican Republic, Mexico and Peru). These countries were selected because they participated in PISA 2015 and TERCE, and because their sampling design permitted the application of the planned analytical techniques (this was not the case in the remaining five countries that also participated in both assessments, namely Argentina, Brazil, Chile, Colombia, and Uruguay). The included countries represent the four main geographical regions in Latin America, respectively Central America, the Caribbean, North America, and South America; however, as mentioned before, they were not deliberately chosen with this or any other consideration in mind.

Data and methods

The analysis draws on publicly available data from UNESCO’s TERCE (grade-based), and OECD’s PISA 2015, (age-based) for the four aforementioned countries. The analytical method used was multiple linear regression; regression slopes were compared by using the z-statistic.

Results

The article concludes that, after adjusting for gender and socio-economic status, the regression slope that represents the achievement gap between repeaters and non-repeaters is approximately twice as large in PISA as in TERCE. This result is observed with small variations in all four countries included in the analysis, and in all three domains covered by both assessments (namely mathematics, reading and science). All observed differences between the assessments are statistically significant at the 95% confidence level.

Significance and policy implications

At the country level, this result has important policy implications: a country may be more or less inclined to revise their policies regarding grade retention depending on whether they participate in one type of assessment and not the other, or whether they pay more attention to the results stemming from one of them. This could also operate as a perverse incentive in two ways: countries could manipulate their retention rates to improve their standing in an ILSA; conversely, they could decide to join an ILSA over another based on their current repetition rates, or on their intentions to change their rates.

On the global level, this could also have important implications: How do these different policy conclusions and recommendations link back to the IOs agendas as shaped by their mandates? Do they reinforce policy beliefs that were already widespread in these organizations?

Author

• Manuel E Cardoso, UNICEF / Teachers College, Columbia University