Better monitoring for better programming: How automatic feedback and decentralized data can improve project implementation

• In Event: ICT4D Practice track II: Learning Design, Analysis, and Evaluation

Proposal

Save the Children's Early Literacy and Math (ELM) and Literacy Boost (LB) programs have a strong base of evidence from rigorous impact evaluations of learning outcomes (Friedlander and Goldenberg, 2016; Dowd et al, 2013). However, the impacts of these programs and the groups for whom these programs work best, often varies by context. It is often difficult to pinpoint the reasons for this variation. Traditional monitoring data does little to help unpack these impact evaluations, emphasizing the quantity of inputs and outputs. Program implementers often lack systematic in-depth knowledge about program quality that could help understand results and improve programs. The fidelity of implementation of these programs is often unknown until too late.

This problem led to the development of a suite of electronic tools to measure the quality and fidelity of implementation of ELM and LB that attempt to bridge this challenge. The goal of these tools is to:

1) collect meaningful data to help better understand the outcome data collected in impact evaluations,

2) immediately deliver feedback to on-the-ground implementers on how to improve program quality, and

3) delivery data to program managers in real-time to aid programmatic decision makers.

The tools consist of digital monitoring forms that allow classroom observers to record the quality of classroom environments and project implementation. These tools automatically tabulate scores and deliver feedback messages to improve areas of weakness. Lastly, data is automatically uploaded onto dashboards that program managers can use to identify trends in quality and allocate additional resources to problematic areas.

Developed in 2016 and 2017, this paper reviews the tools developed under Tech4EPM, the initial data collected with them in 2017, and implications for future programming decisions and improvements.

Dowd et al.: “Literacy Boost: Cross Country Analysis Results” (2013): Do at-risk groups benefit more from Literacy Boost? (Regression evidence suggests “yes.”)


Friedlander & Goldenberg: "Literacy Boost in Rwanda: Impact Evaluation of a Two Year Randomized Control Trial"

Authors

• Jonathan Michael Seiden, Save the Children

• Christine Jonason, Save the Children - USA

• Jane Leer, Save the Children - USA