Theory Based Evaluation in Kenya: Using Research to Inform National Scale Implementation

• In Event: Large-scale Experimental Research on Early Grade Reading: Insights from India, Kenya and South Africa

Proposal

Objectives or purposes
Randomized controlled trials (RCTs) have proliferated in the education sector in developing countries. These studies have contributed to a rapid increase in the rigorous evidence available for policymakers regarding whether particular programs have an impact on learning outcomes. Unfortunately, the science of RCTs in developing countries remains relatively nascent in providing evidence to those policymakers on the causal mechanisms at work within the overall treatment effect. In other words, the RCT tells the education policymaker whether the program works, but not why, how or under what conditions. Responding to this, recent research in Kenya has allowed the Ministry of Education to test the theories that underpin several RCT designs.

Perspective(s) or theoretical framework
Theory based evaluation is a growing field in the international education field. The RCTs in the Tayari program and the Primary Math and Reading (PRIMR) Initiative (the pre-primary and primary education sector, respectively) allowed the MOE to design policy and large-scale interventions based on which program conditions were most successful.

Research methods or modes of inquiry (including data sources, evidence, objects and/or materials)
PRIMR and Tayari utilized RCTs designed for policy decision-making to answer the following questions:
1) Is a 10:1 or 15:1 ratio of coaches to schools more effective and more cost-effective?
2) Do tablets at the student, teacher or coach level have the biggest effect and most cost-effective?
3) Does impact on literacy and numeracy require one or two years to occur?
4) Can pupils learn to read in mother tongue in complex language environments?
5) Does assignment to a mother tongue treatment group have a bigger impact than assignment to a non-mother tongue treatment group?
6) Does assignment to a mother tongue treatment group impact English and Mathematics outcomes more than assignment to a non-mother tongue treatment group?
7) Which combination of teacher training and coaching, student books and teachers’ guides have the biggest effect on learning?
Answering these questions requires the RCT design to go beyond the simple average treatment effect, looking at whether the treatment outperformed the control, to investigating how Kenyan policymakers could and should design large scale implementation efforts.

Results and/or substantiated conclusions or warrants for arguments/point of view
The findings from the studies above have been published in several peer reviewed journal articles. In each of these papers, Kenya has contributed to what is learned in implementing instructional improvement programs within the existing government systems and using their personnel. More importantly, the Kenyan Ministry has expanded the results from an academic exercise to designing national literacy and numeracy programs that respond to the findings from the research.

Scholarly significance, originality and/or creativity of the study or work
The Kenyan Ministry of Education has been at the forefront of utilizing research evidence in designing programs. Initial findings from the Tusome national literacy program, implemented in every public primary school in the country, is positive enough to suggest that the experiment with theory based evaluation has allowed for an effective method of policy design that other countries and contexts should consider.

Authors

• Richard Belio Kipsang, Ministry of Education, Kenya

• Benjamin Piper, RTI International