Making sense of international large-scale assessment: What education actors in two states say about PISA

• In Event: Highlighted session: Critical reflections on the development, use and policy impact of international large-scale assessments in education

Proposal

During the past three decades, as international large-scale assessments (ILSAs) have become more common, a new phenomenon has emerged as sub-national units, such as states and large cities, have chosen to participate in ILSAs above and beyond federal commitments to national participation (Engel & Frizzell, 2015). This growing participation in ILSAs gives sub-national systems access to data, analysis and reporting that is typically only received by the national governments of participating countries. Despite the access to this new internationally comparative performance data, there has been almost no systematic exploration of whether or how this data is used in the service of education policy and practice at the sub-national level.

This paper discusses findings from a project that analyzed whether and how actors from two US states that participated in the Programme for International Student Assessment (PISA) used assessment results, and to what ends. We conducted a comparative case study of two US states that participated in the 2015 cycle of PISA and followed each state’s subsequent use of PISA results. We analyze use at two levels: what assessment products and data (e.g., ranking tables, reports, background questionnaires, released items) are used as well as how various actors in the state spaces use them.

CONCEPTUAL FRAMEWORK
Existing scholarship offers a number of insights into the ways in which policy actors use PISA. Comparative policy scholars have documented the emergence of an ILSA “regime,” documenting both the increasing numbers of countries participating in standardized international assessments (Kamens & McNeely, 2009; Pizmony-Levy, 2013; Meyer & Benavot, 2013) and their motivations for participating (Addey, Sellar, Steiner-Khamsi, Lingard &Verger, 2017; Kijima, 2010). There has also been a focus on impact that participation has on education discourse in particular domestic country settings (Pizmony-Levy, forthcoming; Waldow, Takayama & Sung, 2014; Green Saraisky, 2015). Although scholars have noted that ILSAs research can be used for a variety of policy purposes (Howie & Plomp, 2005; Chabbott & Elliott, 2003), there is little evidence to show that PISA has had any direct impact on education policy.

We also draw on the research and knowledge utilization literature (e.g., Davies & Nutley, 2008; Weiss, 1979). In this literature, there is broad consensus that evidence is rarely used in a linear way, but instead is influenced by a variety of factors, including capacity (at the organizational and individual level) (Honig & Coburn, 2008; Landry et. al., 2003; Massell et.al., 2012; Penuel, et.al., 2018); networks (Goertz, et. al. 2013; Honig, 2004) and context (DuMont, 2015). Education in the U.S. is almost wholly a function of state and local authority (e.g., Carnoy, Garcia & Khavenson, 2015; Manna, 2011) and research has shown that different state contexts result in different education policies and outcomes.

DATA AND METHODS
In order to explore whether and how PISA data and research products are used by sub-national units, we conducted a comparative case study of two states, to provide a detailed account of the conditions under which ILSA research is used and how. These two states were the only two states to participate in the most recent PISA round. By “PISA data” we refer not only to the ranking tables and mean scores of sub-national participants, but also to the assessment instrument itself, accompanying frameworks and reports, and the extensive dataset that is available to participants.

For each case, we identified key interview informants and used snowball sampling techniques for additional informants. In each state, respondents included senior state government officials, philanthropists, business leaders, academics, union leaders and journalists (n=35). We also conducted interviews with informants from the National Center for Educational Statistics (NCES), as well as at the OECD to triangulate state-level data with national and international perspectives.

Additionally, we collected media articles, legislative records, transcripts of hearings, think tank reports, and press releases from both governmental and non-governmental organizations. We analyzed both states across a common set of background characteristics, including such variables as individual capacity, organizational structure, ownership and leadership, and networks. Data were analyzed using NVivo using both open- and theoretical coding approaches.

FINDINGS
Despite the fact that the literature predicts that variation between the states should lead to variation in use, we find that use of PISA in both states looks remarkably similar. We find little, if any, evidence of PISA use that directly influenced education policy or practice. Instead, the rankings and raw scores of PISA results were used in the states for primarily public relations and communications purposes. In fact, some informants had no idea that any information beyond the rankings existed.

These findings suggest the need for additional explanations for PISA use in the states. We suggest that under conditions of uncertainty the notion of sense making (Weick, Sutcliffe, & Obstfeld, 2005, 410) – the ways in which policy actors and education stakeholders understand and ascribe meaning to PISA – also needs to be taken in to account to understand the extent to which PISA results are used to inform policy and practice. It is sense making that allows actors to organize, label, reflect on, and then act on their understandings of an input to create a plausible story about what they have experienced.
We posit that the ways in which state policy actors make sense of PISA shapes their engagement with and use of PISA’s data and other products. Although motivation for participation varied between the states, we find that actors’ perceptions of PISA as a comparative ranking table is what influences use. In the full paper we present and discuss additional evidence that supports these findings.

SIGNIFICANCE
One of the hallmarks of the “education for development” push has been the reliance on numbers and quantitative metrics as the forms of measurement to promote educational progress. However, we find that with regards to the sub-national participation of two US states in PISA has not led to any measurable effects on education policy or practice. We argue that sense making is an underexplored variable in ILSA/policy research and suggest that scholars would do well to begin to study the use angle of ILSA participation.

Authors

• Nancy Green Saraisky, Teachers College, Columbia University

• Oren Pizmony-Levy, Teachers College, Columbia University