Context, mechanisms and outcomes of accountability policies; a comparative review of 14 systems

• In Event: Rethinking Educational Accountability: Autonomy, Evaluation, and Democratic Governance Across Systems

Proposal

Objectives or purposes
This paper presents a study of accountability systems in 14 jurisdictions that considers the contextual and normative nature of education policies in presenting relevant evidence of the functioning and outcomes of these systems. The study was conducted for the recently established Centre for Education Systems in the UK; set up to provide governments, advisors, and campaigners with evidence, and to provide a deep understanding of different policy areas. The Centre’s approach is to provide insights into alternative approaches in key policy areas, explain how policy instruments interact, develop an overall architecture to help build system coherence, and identify intervention priorities. The study we undertook was a review of accountability systems. It is one of the policy areas that served as the proof of concept for the operating model of the Centre, the other being curriculum.

The study is part of a wider trend in evidence-informed practice that has gained significant momentum in the past decade in an attempt to improve learning outcomes. This is particularly evident in the establishment of a wide range of clearinghouses and what works centres that collect and vet all the studies evaluating a given program. By providing an overall rating of their effectiveness (and sometimes costs), the hope is that schools will use it to make more informed decisions to improve their education. When it comes to education policy however, there is no established practice of evidence use, although novel ways of synthesizing evidence for policy relevance continue to emerge, according to Wadhwa and colleagues (2024). Our study and Centre aims to contribute to this emerging field of work.

The study presented here on accountability included a review and synthesis of policy documents and empirical research to answer the following questions for 14 jurisdictions:
1.What is the purpose of accountability?
2.What is the structure of the accountability?
3.How is accountability policy made and introduced?
4.How is accountability policy evaluated?
5.How does context shape accountability policy and reform?
6.What is the evidence about the effects of the accountability system?

Perspective(s) or theoretical framework
The study was informed by a theoretical framework that conceptualizes ‘accountability in education’ with the purpose of informing the analysis of policy documents and empirical studies in the review.

Accountability is a broad concept that can have different meanings, depending on the field of study or the specific scope, functions, and goals chosen in a particular jurisdiction. ‘In its most general sense, accountability means giving a justification of what one has done’ (Hoffer, 2000, p.529). Such justifications can be given in an informal and non-routinized manner, or through more formalized systems. The informal manner involves simply giving an account, taking responsibility for explaining one's actions in a form of social interaction, while the latter implies being called to give an account by an authority who enforces responsibility on another through more standardized and prescribed systems (Mulgan, 2000).

Our focus was on formal types of secondary school accountability where educational institutions have an obligation to report on, explain or justify what they have done to one or more sources of authority, drawing on various types of information. School accountability typically consists of codified standards and measures with certain consequences for reaching or for not reaching the standards. ‘Formal’ refers to the measures used and how information is shared. Consequences however can be both formal (and intended), such as when inspection outcomes are used to allocate support and resources by government agencies, as well as informal (and sometimes unintended), such as when stakeholders act on outcomes in ways unexpected in the original design of the system.

Methods, techniques, or modes of inquiry
The study included 14 jurisdictions: England, Scotland, Wales, Northern Ireland, Ireland, the Netherlands, Poland, Finland, France, Singapore, New Zealand, Ontario, Japan, and Estonia. The choice of 14 jurisdictions was informed by their performance on PISA, as well as contextual differences that may offer policy alternatives and useful comparative insights. Given the limitations in PISA scores’ validity as a measure of success and well-documented concerns about reliability, we have avoided taking an overly rigid, PISA-based view of success. We recognise that countries will always be in flux.

The study was organized in three phases where the first phase included an analysis of policy documents to answer research questions 1-4. These questions were answered through a policy document analysis and fact-checking exercise with policy experts. They provided descriptive detail about each country’s accountability system, including the wider cultural and economic context in which these systems operate.

The second phase focused on research question 5 and 6 through a rapid systematic review of empirical studies from each country.

These thematic reports were then analysed in the third phase for a description of context, mechanisms and outcomes of accountability in each country. This realist frame allowed us to understand the generative mechanisms that shape outcomes of accountability and provide a nuanced understanding of how and why specific accountability elements or types of systems work in specific contexts.

Results and/or substantiated conclusions or warrants for arguments/point of view
Our study provides a comparative descriptive analysis of accountability systems within 14 jurisdictions. It sought to identify similarities and difference in these systems to understand if, how, and when, these lead to intended and unintended outcomes. Across the fourteen jurisdictions we find a set of common ‘context, mechanism, outcomes’- configurations that include.
- Formal and informal pressure to improve (from stakeholders using accountability outcomes and consequences from performing below targets)
- Professional development and normative pressure to meet accountability standards and targets
- Accepting and using feedback from inspections and/or standardised assessments
- Capacity-building and establishing networks and cultures for evaluation and the use of performance data.

Author

• melanie Ehren, Vrije Universiteit Amsterdam