Repetition rising? Using PISA to track its alleged post-pandemic surge and repurposing in high- and middle-income countries

• In Event: Highlighted Session: Methodological Challenges with latent constructs

Proposal

Historically, repetition is higher in low-income countries. This is often deemed an internal inefficiency issue by some economists of education (Haddad 1979), sometimes attributed to the limited capacity these countries allegedly have to manage their own education systems. More recently, scholars like Brophy (2006) have acknowledged that, especially in low-income settings, repetition does not always mean retention, and that it is not always initiated by the school, but sometimes by the students’ own families. Moreover, Agasisti and Cordero (2015) find that variations in repetition rates across countries may be partly due to cultural factors.

Recently, some anecdotal evidence from high- and middle-income countries points to an increase and perhaps a qualitative change during the pandemic. This paper uses quantitative evidence to explore the following questions:

1. Was the pandemic associated with a repetition surge in these settings?
2. Did repeaters’ sociodemographic profile change during the pandemic?
3. Did the association between repetition and learning outcomes change during the pandemic?

One initial challenge is the unavailability of recent globally comparable data. The advent of the SDGs and their unprecedented number of education indicators under Goal 4 has led to deprioritizing other “legacy” indicators, such as repetition rates. This, coupled with the pandemic itself, has led to a precipitous decline in the number of countries reporting repetition rates: 122 countries reported this indicator in 2016; but by 2019, still before the pandemic, it was down to 95; in 2020 they were 55; and in 2021, only five countries.

Meanwhile, a related SDG indicator like 4.1.5, “Percentage of children over-age for grade”, may not be useful in our current post-pandemic situation, since it is operationalized, for instance, as the “Percentage of pupils enrolled in lower secondary general education who are at least 2 years over-age for their current grade, both sexes (%)” (my emphasis). Given the nature of the disruption, a substantial number of students may have been affected, including many never previously retained, but the typical impact is likely lower than two years of schooling.

Simultaneously, the nature and function of grade repetition may have changed during the pandemic, particularly in high- and middle-income countries, and especially in those where repetition was historically low. Brophy (2006) has argued that, especially in developing countries, repetition does not necessarily equal retention, and it is not always initiated by the school, but by the student’s own family. An alleged surge of repetition in countries like the US and Canada has been accompanied by some anecdotal evidence of parents initiating the repetition decision (https://learningenglish.voanews.com/a/school-systems-see-more-students-repeating-grades/6733561.html; https://www.usnews.com/news/us/articles/2022-09-01/more-kids-are-repeating-a-grade-is-it-good-for-them). These changes may have affected, perhaps in a transitory manner, the relationship historically observed between repetition and academic achievement.

There are at least three challenges when studying the relationship between repetition and achievement: rarity, endogeneity, and the comparison strategy. First, even in times of crisis, repetition remains relatively rare; this complicates finding statistically significant associations in sample-based studies, given the small number of repeaters (Cadigan et al., 1988). Secondly, on the other hand, the association between repetition and achievement is affected by endogeneity, or reverse causality, because repetition itself is typically a remedial measure applied selectively to low-performing students; this results, in cross-sectional studies that do not control for prior performance, in a substantial overestimation of the negative “effect” of repetition on achievement (Jackson, 1975; Alexander et al., 2003). Finally, the comparison strategy also affects this relationship: it matters whether the comparison between repeaters and non-repeaters is age-based (as in PISA) or grade-based (Allen et al., 2009; author, 2020).

These three issues will be “solved” by applying the same strategy, namely by using two successive rounds of PISA data: one from 2018, pre-pandemic; and the latest, from 2022, post-pandemic. It has been shown (Author, 2020) that PISA overestimates the gap between repeaters and non-repeaters because it does not control for prior achievement, leading to endogeneity issues, and because its comparison strategy (age-based) also inflates the gap. In that regard, PISA may be deemed a magnifying glass that finds statistically significant associations despite the small number of repeaters. This could be problematic if we were drawing absolute inferences about this association. But what we are focusing on here is the evolution of this association in the intervening time between before and after the pandemic. Therefore, the endogeneity and comparison strategy issues become less relevant.

Analytical challenges and solutions aside, this raises one interesting paradox. PISA, like all ILSAs, shows substantial methodological limitations when trying to answer any specific research question, because their designs were not optimized for any particular research purpose (Rutkowski, 2016), and especially not for a policy or practice so challenging to evaluate as grade retention (Author, 2022). Therefore, to avoid all these challenges, it is best to compare PISA only to itself, which paradoxically may generate some interesting path dependencies: it is not necessarily PISA’s methodological strengths that make it so formidable; it may also be its weaknesses.

In order to answer the first question, on the prevalence if repetition before and after the pandemic, I will use a fixed effects regression model (or random effects, depending on which model’s assumptions are met) with data from the two aforementioned PISA cycles, for the more than 50 countries that participated in both.

For the second question, about repeaters’ sociodemographic profile, I will perform a logistic regression of students’ repetition in lower secondary on sociodemographic variables including gender (boys are more likely to repeat) and socioeconomic status, accompanied by interaction terms for countries.

Finally, I will regress achievement on repetition and the aforementioned sociodemographic variables as controls, and countries as interaction terms. Because the major domains for PISA 2018 and 2022 are, respectively, reading and mathematics, I will, perhaps also paradoxically, use science as the achievement domain, to avoid differences in measurement error that may affect comparisons.

This paper contributes to the field by using what has over time become a familiar tool, namely PISA, in an innovative way, in order to face the policy challenges of an unprecedented situation.

Author

• Manuel E Cardoso, Teachers College, Columbia University