Why do some international assessments succeed, and others do not?

• In Event: Comparative Case Studies: New Designs, New Directions--Part 2

Proposal

International large-scale assessments (ILSAs) are considered a success story. The past two decades witnessed an immense growth in the number of international and regional assessments and in country participation in these assessments (Kamens & McNeely, 2009). Scholars disagree on the relationship between country performance on ILSAs and economic development, which it the main socio-logic behind the ILSAs regime (Hanushek & Woessmann, 2010; Ramirez et. al, 2006). Nevertheless, scholars have documented the ways in which these assessments shape public discourse, public opinion, and educational policy (Green Saraisky, 2015; Pizmony-Levy, 2018; Pizmony-Levy & Bjorklund, 2018; Waldow, Takayama, & Sung, 2014).

This perspective on ILSAs, however, is not complete as it focuses on successful cases. That is, current scholarship on ILSAs overlooks failed attempts to launch ILSAs. Drawing on organizational theories and innovations studies (Binder, 2000; Fullan, 2001; Rogers, 2010), I seek to address the following research question: Why do some international assessments succeed, and others do not? Based on archival research and interviews, I examine the extent to which different factors explain successful institutionalization of ILSAs (i.e., long term and sustainable administration). I compare three organizations that initiated and administrated ILSAs: (1) the US-based Education Testing Service (ETS) and its International Assessment of Educational Progress (IAEP); (2) the International Association for the Evaluation of Educational Achievement (IEA) and its Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS); and (3) the Organisation for Economic Co-operation and Development (OECD) and its Pro-gram for International Student Assessment (PISA). Although these three organizations have offered similar technical solution to the growing demand for achievement data in key subjects, they achieved different outcomes with respect to sustainability. While assessments by the IEA and the OECD were institutionalized into regular cycles, the ETS-IAEP was abended after two cycles.

Trained as a sociologist, I first approached the research question with a comparative-historical methodology (Mahoney, 2004; Ragin and Becker, 1992). I assumed the three cases are independent from each other and leveraged the variation in the outcome (two success cases and one failed case). Two factors have contributed to the failure of the ETS-IAEP. The first factor is “buy-in” from participating countries. In contrast to assessments by IEA and OECD, ETS was not able to recruit diverse set of countries to participate in its first two rounds of IAEP. The second, and related, factor is “international optics.” Both IEA and OECD are international organizations that bring together representatives from different countries. The ETS is a US-based non-for-profit that sought to export the National Assessment of Educational Progress (NAEP) to the world. Lacking the international optics and legitimacy, ETS-IAEP failed to institutionalize.

The story, however, is more complicated. In interviews with ILSAs leaders, I heard several times that key figures in the (failed) ETS-IAEP later moved to IEA and made a significant contribution to the institutionalization (or success) of TIMSS and PIRLS. In other words, the two cases are dependent on each other or at least connected. Thus, I approached the research question with comparative case studies (CCS) framework (Bartlett and Vavrus, 2017). Specifically, I followed the transversal axis to illustrate the ways in which the US government was able to reconfigure the field of ILSAs and to reorient the field from research/science to policy/accountability. The change took place in two steps. First, the US government introduced the ETS-IAEP as a competition to the IEA. Second, assessments professionals and technology developed at the ETS-IAEP heavily influenced the IEA’s practice and assessment.

Author

• Oren Pizmony-Levy, Teachers College, Columbia University