Is a RCT appropriate for evaluating Ethiopia’s large-scale quality education reform program?

• In Event: What contributions can randomized experiments (RCTs) make to our understanding of education systems reforms?

Proposal

Improving students’ learning outcomes is central to the Ethiopian’s governments large-scale education reform program the General Education Quality Improvement Programme for Equity (GEQIP-E) (2017-2022). RISE Ethiopia aims to assess the impact of GEQIP-E on equitable learning in Ethiopia over the next five years. Randomized control trials (RCTs) are increasingly adopted as a way to provide answers on what works best in particular contexts. Evidence from successful RCTs enables subsequent decision making about the effectiveness of particular interventions. A critical question remains about the applicability of findings from RCTs to other contexts or to different conditions. A further question relates to the feasibility of implementing an RCT in a constantly evolving, real life, large scale, reform process, of the kind of GEQIP-E. There are also political considerations in undertaking RCTs, which may not always be palatable to governments.

While acknowledging the important contribution that RCTs make to the empirical field, this paper interrogates the utility of RCTs in the context of undertaking systems research in Ethiopia with respect to assessing the impact of the GEQIP-E. Many education RCTs focus on ‘single component’ interventions. More recently, some studies have evaluated the effect of combining two reforms, showing that single interventions have more limited impact. This work, some of which has been done to very high scientific standards, has been undertaken in conditions where the researcher has considerable control over the provision of the ‘treatment’. By contrast, there is less research focusing on bundles of government-led interventions, which are more consistent with the real world of education reform where governments put in place several reforms simultaneously. Therefore, there is a gap in the empirical literature about the effectiveness of the provision of these bundles at scale. Assessing the impact of GEQIP-E reforms is an opportunity to do so.

Historically, there has been little systematic research in Ethiopia on the impact of past education reforms driven in large part by government resistance to such explorations. In this context, any research approach to understanding system reforms has to be done in close collaboration with the Ministry of Education who value a holistic assessment of GEQIP-E. This approach of the assessment of GEQIP needs to include an evaluation of the bundled set of interventions under GEQIP-E, and then the exploration of the implementation of specific elements of that package, an approach which is difficult to undertake using RCTs alone.

More specifically, RCTs require some form of variation from the status quo in the implementation for comparability purposes. This is a challenge for understanding and evaluating GEQIP-E, given the complexity of this program in both design (for example, different results areas such as efficiency, quality, equity, systems strengthening) and implementation (for instance, challenge in ensuring fidelity of implementation, stakeholder capacity, allocation of resources). In this context, a pure RCT approach is not feasible. The paper will outline the approach to be adopted by the RISE Ethiopia team that aims to allow us to identify the impact of GEQIP reforms over time and across locations.

Authors

• Tassew Woldehanna, Addis Ababa University

• John Hoddinott, Cornell University

• Ricardo Sabates, University of Cambridge