Phone-Based Learning Assessments and a Rapid Randomized Trial of a “Low-Tech” Phone and SMS Intervention in Botswana

• In Event: Using Evidence to Inform Education Delivery during COVID-19

Proposal

Assessing children and youth remotely is essential to mitigating the adverse short and long-term public health and education impacts of the COVID-19 pandemic, as well as future school closures due to health and other crises. While there is existing literature on best-practice strategies to carry out phone-based surveys to adults, on oral face-to-face testing of learning among children and youth, and on using technology to help community health workers identify ill or at-risk children, there is little evidence on assessing learning among children and youth over the phone.

An educational NGO partnered with the Ministry of Basic Education in Botswana to collect over 10,000 phone numbers in schools in four out of ten regions in Botswana before schools closed for lockdown. Since schools have been closed, caregivers and students have been contacted to participate in remote learning interventions rolled out as a randomized evaluation. The presentation will discuss the results of the evaluation and practical lessons that the NGO, researchers, and Ministry have learned while conducting phone-based assessments on the educational delivery methods, and how these lessons can inform efforts to evaluate different methods full-time. Using phone numbers collected from primary schools for students in grades 3-5, the research team identified 4,550 households who were willing to receive remote learning support via mobile phone. Each household was randomly assigned to one of three groups:

1) Weekly SMS message only: Households in this group received a weekly SMS containing several simple math problems (“problem of the week”).

2) Weekly SMS message + phone call: In addition to the SMS messages, households in this group also received a 15-20-minute phone call, in which facilitators answered any questions related to the “problem of the week” and provided additional practice questions. The calls required both parents and children to be present, as a way to encourage engagement and maintain accountability.

3) Comparison group: Households in this group did not receive any type of phone learning support.

After four weeks, facilitators conducted phone surveys with parents and students from half of all households to collect information on their engagement in the educational activities, parents’ perceptions of their child’s learning, and student learning outcomes, as measured through standardized test scores.

After week four, researchers randomly selected a subset of the households from the SMS only and SMS + phone groups to receive an additional support program: targeted instruction. In this subsequent intervention, which is ongoing, students receive text messages tailored to their individual learning levels over a period of four weeks. For example, students who know addition will receive subtraction problems to push them to a higher level of learning, whereas students who know multiplication will be sent division problems. Through weekly phone surveys, facilitators will collect students’ answers to the “problem of the week,” which will be used to target subsequent weekly SMS messages.

Both the SMS-only and SMS + phone calls interventions had positive impacts on students’ learning, particularly for lower-performing students. The programs also led to higher parental engagement in educational activities, and improved the accuracy of parents’ perceptions of their child’s learning level. This trial provides some of the first experimental evidence on minimizing the fallout of the pandemic on learning.

The phone-based assessments were administered to 2,250 students who were in grades 3-5 before schools shut down by over 70 former teacher aides (“assessors”) who call households directly. Data from these phone assessments was complemented with data from face-to-face assessments before schools were shut down for the pandemic. These data informed preliminary practical lessons for future phone-based assessments, notably the importance of specifically protecting children (do no harm protocols), two-step validity testing of the instrument, and utilizing multiple methods to encourage participants to answer the phone (such as text alerts coupled with phone calls). This presentation will discuss the implications of the results of the evaluation, the importance of incorporating student feedback and experiences as direct stakeholders during school closures, and how education delivery methods can accordingly be adjusted.

Author

• Noam Angrist, Young 1ove