Making measures matter: Criteria for ensuring the impact of cross-national assessments

• In Event: Highlighted session: Critical reflections on the development, use and policy impact of international large-scale assessments in education

Proposal

How is the topic relevant to comparative and international education?

This paper is an attempt to sort through selected literature on international survey work on literacy to understand the potential of these surveys to contribute to the creation of sustainable educational systems in developing and developed nations. The increasing presence and influence of international surveys (Wagner et al. 2012) makes it imperative that academics and policymakers can comprehend the results and appropriately understand the applications of such results. However, we are in danger of a tyranny of technicism, where data becomes the primary concern in decision-making without any consideration of philosophical commitments or the cultural settings of education (Sobe, 2018). Sustainable development requires that goals and measures reflect the conditions and priorities of developing economies, and the ability to engage critically with these structures is key in ensuring that those leading developing systems can shape their work.

How does the contribution build on existing work?

The literature on literacy assessments is fragmented and often partisan; authors tend to be strong supporters of international testing or strong opponents (cf. Dubeck & Gove, 2015). A more nuanced and responsible approach is needed. The aim of this paper is to begin identifying key questions or issues that to take into account when approaching the survey process and the results generated.
Two existing surveys will be used to anchor the discussion. The first is the Early Grade Reading Assessment (EGRA), primarily focused on understanding the emergent reading competencies of children in divergent cultural and linguistic settings. EGRA works by increasing the capacity within the individual country to create assessments appropriate for its own context (cf. Dubeck & Gove, 2015). The second is the Program for International Assessment of Adult Competencies (PIAAC) (OECD, 2018). The is the latest iteration of a series of surveys intending to capture competencies as a proxy for human capital at the national level, permitting comparison of skills stocks in different nations (St. Clair, 2013). These two instruments offer powerful and complementary insights into the uses of literacy surveys.

Five initial aspects to consider are:

Clarity of objectives
An assessment may be designed to provide internationally comparative data, system performance information, school evaluations or inclusive data on every student’s performance (Wagner et al. 2012). The choice of objective affects sampling, cost, statistical model, model of literacy and almost every other dimension of the assessment. There is a tendency for assessments to demonstrate considerable drift. EGRA was designed to be highly customizable to social and linguistic context, meaning that international comparability was severely compromised. Yet recently comparative data has started to be generated, initially around proportions of zero scores in different countries (Dubeck & Gove, 2015). PIAAC, which was explicitly designed to generate comparable information, has started to design assessments for individual learners (St. Clair, 2016). These inevitably challenge the legitimacy of the tools.

Unanticipated results
In order to justify resources consumed, unexpected results are an important indicator that real-life phenomena are being measured, and the assessment adds value. If assessments consistently give results entirely in line with the expectations of common sense they add little to the sum of human knowledge, but an unexpected result can raise important issues for further exploration. For example, in the UK PIAAC the age group who had just left school scored lower for competencies than the group a few years older. Between 16 and 24 skills were lower than older cohorts irrespective of the educational status of these young people (BIS, 2013). This suggests early years of employment may play a crucial role in consolidating competencies—an unexpected and valuable insight.

National utility
Given that in many cases there is considerable cost to the nation-state in participating in international assessments, it seems reasonable to expect that there will be a benefit at this level. The evidence of such benefits remains partial and anecdotal. One of the primary rationales for international assessments (and the related comparisons) is that it can provide policyholders with both an argument and guidance for educational reform. There is a tension between the trans-national framing of assessment to permit comparison and the contextualized approach likely to give rise to fully implementable insights.

Educational nativism
Those not coming from a North American background may well be struck by the growing influence of the Kindergarten to Grade 12 concept of education. This impinges upon other schooling designs and upon adult and alternative learning, required to categorise capabilities against a notional 12 grade scale. One example of the disjuncture this causes is systems that start the academic portion of schooling at a later age being considered less effective. Another is that a mature person with limited literacy may be represented as functionally equivalent to a seven year old, which is fundamentally problematic.

Reliability and validity
Consideration of these aspects goes beyond the internal consistency of results (though they are not always highly internally consistent, or tested for consistency). A more challenging and deeper consideration is whether the assessment is meaningful. Score differences are, by themselves, not information. There has to be a framing concept of theory to turn data into information. For example, a highly limited model of literacy undermines an assessment’s claim to real world value however well the survey is designed and implemented (St. Clair, 2013).

How is the contribution situated?
While relevant to many contexts, the questions raised are intended to be especially pertinent to developing economies.

What are the implications of the critique?
Consumers and producers of international education assessments need to be sensitive to the claims made and the grounds upon which they are made. It is crucial that educators and policymakers fully understand the promise and the limits of international assessments in order to maximize their value.

How original is the contribution/ what do we learn?
The overall aim is to support informed engagement with educational data, and this paper promotes the possibility of actively formulating ways to interrogate that data. In doing so, I hope to offer a pathway to more fully informed decision-making at every level.

Author

• Ralf St.Clair, University of Victoria