Combining Experimental Methods and Causal Chain Analysis to Investigate Education System Reform in the Learning at Scale Project

• In Event: Innovations in research methods to study education systems

Proposal

1. Context

Learning outcomes are poor and instruction is ineffective in many low- and middle-income countries. Although the number of successful programs is growing, few have demonstrated impact at scale. The Learning at Scale project, funded by the Bill and Melinda Gates Foundation, identified 8 existing projects, in 7 countries, that have demonstrated impact on literacy skills at scale. Research was conducted on the 8 projects to address the following questions: (1) What classroom ingredients (teaching practices, classroom environment) lead to learning in projects that are effective at scale? (2) What methods of training and support lead to teachers adopting effective classroom practices? (3) What system support is required to deliver effective training and support to teachers and to promote effective classroom practices?

2. Theoretical framework

The theory of change consists of causal chains of effect from ministry to district to school, based on the “bare-bones” model of system functioning including: (1) Setting expectations for the outcomes of education (2) Monitoring and holding schools accountable for meeting those expectations (3) Intervening to support the students and schools that are struggling, and holding the system accountable for delivering that support.


3. Mode of inquiry, approach, methods, analysis

It is challenging to identify the impact of system reform on learning outcomes. We used a combination of three approaches: (1) Counterfactual logic using quasi-experimental methods (Shadish, Cook and Campbell, 2002) to compare learning outcomes and classroom practices between schools included in the reform process and comparison schools; (2) A mechanistic account of the causal chain (Kneale et al, 2018) developed through qualitative and quantitative data; (3) A structured cross-country analysis (Bennet, Fairfield and Soifer 2019) to integrate findings from Methods #1 and #2.

4. Findings, key learnings, arguments, recommendations

Findings and recommendations focus on the effectiveness of methods in addressing the research questions. We discuss the challenges, benefits and limitations of investigating system functioning through causal chain analysis. The strongest findings from the study are produced when counterfactual and mechanistic approaches align and when similar results are found across countries.


5. Originality, contribution to education debate, possible application

The use of causal chain analysis – especially when combining qualitative and quantitative methods – is relatively uncommon in international education research. Its use, particularly in conjunction with experimental methods, has the potential to advance the study of education systems and may have wide use in the field of education research more generally.

Authors

• Matthew Jukes, RTI International

• Benjamin Piper, RTI International

• Jonathan Stern, RTI International