Using back-end app usage data to understand curriculum implementation and explain programme effectiveness

• In Event: Digital possibilities for improving and evaluating early grade reading: Insights from a series of randomised experiments in South Africa

Proposal

Structured learning programmes are amongst the most impactful education interventions, according to several systematic reviews. These programmes typically provide lesson plans, learning materials and professional support to teachers, all aimed at improving curriculum delivery by teachers. The impact of such programmes surely depends critically on the fidelity of programme implementation by teachers in their daily practice. Yet observing the intermediate outcome of curriculum implementation is difficult. Sometimes work done by pupils in exercise books is used as a proxy for curriculum coverage; sometimes researchers observe a sample of lessons to assess fidelity to the intended pedagogy. Unfortunately, these methods have their limitations, such as changes in behaviour in response to being observed.

In this paper, we exploit back-end user data from an application containing daily lesson plans provided to teachers on an electronic tablet as part of South Africa’s Early Grade Reading Study. The data reveals how often, when and for how long teachers accessed lesson plans across 2022 and 2023. This provides a unique opportunity to gain new insights into the extent of curriculum coverage and how programme implementation varied across teachers and contexts.

There are several noteworthy findings. Firstly, curriculum implementation becomes weaker later in the week (Monday’s lessons are more likely to be accessed than Friday’s), later in each term (Week 1 and 2 lessons are more widely accessed than Week 9 and 10 lessons), and later in the Year (Term 1 and 2 lessons are more widely accessed than Term 4 lessons). This is in line with global evidence of over-ambitious and packed curricula, which teachers struggle to complete, especially in contexts where significant amounts of teaching time are lost due to competing activities. This pattern also implies that the technological aptitude of teachers was not a significant barrier to use, something that was confirmed by qualitative case studies.

Linking the back-end app usage data to teacher questionnaire data, we then compare the patterns in the back-end data to more traditional ways of measuring implementation and curriculum coverage, such as teacher self-reports and observations of students’ workbooks. We also explore how teacher characteristics predict curriculum coverage according to the app-usage data.

Further insights emerge because of an experimental research design: All schools in the study received the electronic lesson plans but some schools were randomly assigned to receive on-site coaching, an intervention which did lead to improved reading outcomes amongst pupils. This allows us to extend the literature on coaching by exploring the intermediate impacts on curriculum coverage.

Looking to the future, this paper points to promising opportunities for using new technologies, and the associated zero-cost back-end data, to better understand the role of implementation in explaining impact evaluation results.

Authors

• STEPHEN G TAYLOR, Department of Basic Education

• Kolosa Nonkenge, Department of Basic Education

• Nompumelelo L Mohohlwane, Department of Basic Education