Innovations in research methods to study education systems

Group Submission Type: Formal Panel Session

Proposal

A systems approach to education reform is critical for improving any nation’s learning outcomes. A system approach takes into account the incentives of actors throughout the education system – from ministries to districts to schools and communities – and how those incentives align, or fail to align, with the goal of improved learning.

Models of education systems emphasize accountability relationships between system actors (Pritchett, 2015). The accountability relationship involves delegation, financing, information and motivation and are many and complex in a system with a large number of actors. The “bare-bones” model of system functioning (Crouch & DeStefano, 2016) attempts to simplify the model by focusing on three elements: (1) Setting expectations for the outcomes of education (2) Monitoring and holding schools accountable for meeting those expectations (3) Intervening to support the students and schools that are struggling, and holding the system accountable for delivering that support.

A systems approach is also holistic in nature. It considers the overall performance of a system in meeting its goals. A key concept is system coherence – how the many accountability relationships reinforce or detract from the goals of an education system. The focus on whole systems presents challenges to the evaluation of reform efforts. Rigorous methods, such as randomized controlled trials, cannot be used to identify the causal impact of holistic system reform. However, it is critical to conduct evaluations to understand the effectiveness of reform efforts. In this panel we present four papers from three project, each presenting an innovative blend of research methods to address this problem.

The Projects

Two papers are presented from the Research on Improving Systems of Education (RISE) Programme. This program is a global research endeavour that seeks to understand how education systems in developing countries can overcome the learning crisis. It includes research in 7 countries: Ethiopia, India, Indonesia, Nigeria, Pakistan, Tanzania and Vietnam.

Lant Pritchett and Marla Spivack will present an overview of the various methodological approaches used in the research from the programme. Ricardo Sabates, Pauline Rose and Tassew Woldehanna from the Ethiopia research team will discuss how their methods have been able to adapt to the COVID-19 pandemic and continue to provide useful information for policy makers.

Another paper represents the Learning at Scale research programme. This programme is examining 8 projects that have been effective in improving literacy outcomes on a large scale. The projects are based in 7 countries: India (2 projects), Pakistan, Kenya, Tanzania, Ghana, Nigeria and Senegal. The Learning at Scale Programme aims to understand the instructional practices that lead to improved literacy outcomes; the approaches to teacher professional development and coaching that develop and sustain those instructional practices; and the ways in which education systems support teachers and their professional development to sustain learning outcomes at scale. Matthew Jukes, Ben Piper and Jonathan Stern will present on overview of the research methods used in the Learning at Scale Programme.

The final paper, by Melanie Ehren, describes a project working in the primary school system in South Africa, including the hierarchy of district, province and national government. The project aims to investigate the large achievement gaps between schools in different poverty quintiles in South Africa.

The Methods
The four papers in this panel present illustrate the importance of using a wide range of research methods to understand the performance of education systems. Broadly, these methods include

1) Theoretically informed descriptions of elements of the education system
2) Studies to identify the determinants of learning outcomes (and the contextual determinants of determinants of learning outcomes).
3) Experimental studies to evaluate components of the education reform process (e.g. the impact of improved accountability on teacher performance)
4) Experimental studies to evaluate the overall impact of system reform – this is possible when reform processes take place in some regions of the country and not others
5) Causal chain analyses (Kneale et al, 2018). This approach involves mapping out the causal relationships between actors in a system and developing methods to examine them. There are two different examples of this approach in the panel. In the Learning at Scale study, a mixed methods approach is used to test accountability relationships between different levels of the system from ministry to schools. The paper from South Africa, similarly maps out causal loop diagrams with teachers, curriculum managers, subject advisors and practitioners/researchers working with schools and districts. The causal loop approach involves a dynamic view of systems, allowing for evolving reciprocal relationships between system actors.


Together the panel present innovative methods and demonstrate how complex systems can be understood through a combination of methodological approaches. The findings have implications for approaches to understanding education systems and education research methods more broadly.


Crouch, L., & Destefano, J. (2015). A Practical Approach to In-Country Systems Research. Paper prepared for the Research on Improving Systems of Education (RISE) Program Launch Conference, Washington, DC, June 18–19, 2015. 1. June, 1–39.

Kneale, D., Thomas, J., Bangpan, M., Shemilt, I., Waddington, H., & Gough, D. (2018). Causal chain analysis in systematic reviews of international development interventions. London: Centre of Excellence for Development Impact and Learning.

Pritchett, L. (2015). Creating Education Systems Coherent for Learning Outcomes : Making the Transition from Schooling to Learning. Paper prepared for the Research on Improving Systems of Education (RISE) Program Launch Conference, Washington, DC, June 18–19, 2015. 1. June, 1–39.

Sub Unit

• General Pool

Chair

• Karen Mundy, Toronto University

Individual Presentations

• Understanding the functioning of education systems: perspectives from the RISE Programme - Lant Pritchett, Blavatnik School of Government, University of Oxford; Marla Spivack, RISE Programme

• How resilient and flexible are education system research methods during a pandemic? The case of RISE Ethiopia - Ricardo Sabates, University of Cambridge; Pauline Rose, University of Cambridge; Tassew Woldehanna, Addis Ababa University

• Combining Experimental Methods and Causal Chain Analysis to Investigate Education System Reform in the Learning at Scale Project - Matthew Jukes, RTI International; Benjamin Piper, RTI International; Jonathan Stern, RTI International

• Causal loop diagrams to explain performance gaps in South Africa’s primary education system - Melanie Ehren, Free University Amsterdam

Discussant

• Laura Savage, DFID