Randomized Controlled Trials of Multimedia Education: The Case of MOBY Multimedia Science Education Impact Evaluation

• In Event: Adapting Distance Education in Resources Constrained Contexts: Lessons from a Multimedia Distance Science Education Project for Girls in Afghanistan

Proposal

This impact evaluation aimed to assess the effects of Lapis/Moby’s Senf mass media programming on secondary school students in Afghanistan, focusing on girls' access to education and social-emotional development. The study employed a randomized controlled trial (RCT) design, running from March to July 2024, with interventions including encouragement to access televised science lessons and supplemental WhatsApp-based tutoring for girls. The evaluation sought to answer key research questions related to access, learning quality, non-cognitive outcomes, and the cost of improving educational engagement.
Our findings show that children who regularly tuned into the broadcasts significantly improved their learning outcomes, measured by test scores. The interventions focused on increasing regular viewership through text message encouragements and tutoring, with both strategies yielding substantial gains in content knowledge over the course of the project. Those who received text encouragements alone saw an increase of about 31 percentage points in their test scores, rising from an initial score of 24% to 55%. This improvement was equivalent to doubling the number of correct answers on an 18-question test. When tutoring was added to regular tuning in, the impact was even greater. Students who combined both interventions improved their test scores by an average of 42 percentage points, raising their scores from 21% to 63%. This powerful combination of broadcasts and tutoring underscores the potential of mixed delivery methods in remote learning settings.
In addition to academic outcomes, we also explored the impact of the interventions on social-emotional learning (SEL). Although the gains in SEL were more modest, with an approximate 9-point increase on the SEL scale, the regular engagement with broadcasts did demonstrate a positive influence. However, attending tutoring sessions did not show a significant additional benefit in terms of SEL outcomes. These findings suggest that while the primary educational interventions focused on cognitive skills, there may be potential for further exploration of how distance learning can impact emotional and psychological development.
One unexpected yet critical finding was that girls outperformed boys in baseline assessments of biology, chemistry, and math, despite facing more barriers to formal education. The tutoring component, which was specifically targeted toward girls, resulted in significant learning gains, not just for girls, but also for boys in the same households. Boys indirectly benefited from the increased exposure to educational broadcasts through the presence of tutoring, highlighting the broader positive effects of household-level interventions. This finding emphasizes the importance of targeting girls in educational interventions while also recognizing the broader community impact.
Although not a replacement for in-class learning, our findings underscore the transformative potential of combining broadcast media with targeted interventions like remote tutoring. The evidence demonstrates that even in conflict-affected and low-resource environments, children can achieve meaningful learning gains through distance education programs. These insights provide a roadmap for scaling up similar initiatives, particularly in settings where formal schooling is inaccessible, ensuring that more children—especially girls—can access quality education, even in the most challenging circumstances.

Authors

• Abdul Hamid Hatsaandh, New York University

• Dana Burde, New York University

• Mohammad Jawed Nazari, New York University

• Heddy Lahmann, New York University

• Rena Deitz, New York University

• Hamidullah Gharibzada, New York University