Assessment of Learning: Evidence from Studies in Six Countries

• In Event: Perspectives on Measuring Learning Remotely: Evidence from and Reflections on Four Different Remote Learning Assessments

Proposal

Global lockdowns caused by COVID-19 demonstrated a renewed need for widespread methods to reach and assess learners in remote settings. Even prior to the pandemic, learning assessments were often costly and occurred too infrequently to inspire relevant educational support.

To fill this gap, we developed and tested a method for conducting remote, phone-based assessments to monitor student learning in basic numeracy operations and applied it to a remote education program that supported student learning during school closures.

The program used a simple delivery model that provides enrolled households with simple SMSes and short, 20-minute phone calls to deliver basic numeracy instruction. Remote phone assessments conducted in four-week intervals determined student learning levels at baseline, midline and at the conclusion of program implementation. These assessments were ASER-inspired and consisted of a simple progression of four maths problems for students to solve - one for each of the four basic operations.

To ensure accuracy, we implemented a series of validity checks to scrutinise the method. These include comparing in-person learning assessments with phone assessment in the same randomised study, implementing back check calls to a random subsample of students, randomising maths problems that students receive during the assessment, and including additional indicators to measure effort and higher-order cognitive skills.

We have since tested phone-based learning assessments in five additional studies, each in a different country. In total, we have assessed over 20,000 students in a period of 18 months using a range of local languages and with a variety of enumerators. In most cases, teachers were able to successfully implement phone assessments.

The scope and success of these efforts contest the perception that learning assessments are necessarily costly and conducted too infrequently. For example, Assessments conducted in our Botswana study cost roughly $4.40 per child compared with the latest Progress in International Reading Literacy Study (PIRLS) that carried an average cost of USD 62.50 per child. Such a low cost and easily accessible delivery model allow for phone assessments to occur at rapid cycles within a span of weeks rather than learning assessments that require a heavier investment and may occur only 2-3 times per decade.

There are several key insights to glean from recent innovations to phone-assessments for learning. First, a strong set of implementing protocols is equally as critical to the success of any phone assessment as is technical validation of learning constructs. Ensuring manageable calling loads, outlining clear expectations for the number of calling attempts per household and establishing a uniform set of in-survey guidelines help to ensure the quality of data collection. Second, there remains a need for additional validation and reliability testing. This includes further efforts to implement phone assessments in a wider range of environmental and educational contexts. Finally, take-up will always be a priority issue and requires careful consideration to maximise the percentage of respondents participating in a phone-based assessment.

Although more must be done, phone-based assessments offer a promising opportunity to complement existing testing infrastructure with higher frequency, low-stakes diagnostics.

Authors

• Colin Crossley, Youth Impact

• Noam Angrist, Youth Impact