Evaluating learner mathematics outcomes in an RCT in South Africa

• In Event: New approaches for measuring and comparing numeracy education outcomes across countries

Proposal

In South Africa, echoing international trends, investment in early grade mathematics research was slower to gain attention. This is now changing, and both the national department of education and other institutions have set in motion Early Grade Math (EGM) research plans. The focus of this presentation, one mathematics programme is currently active in three provinces in South Africa and two pilots are being conducted as randomized control trials (RCT). Evaluation instruments are a critical component of any RCT study. This presentation reports on a validation process of the instruments in use in the pilot in the Limpopo province. The impact evaluation in Limpopo uses as RCT with 120 no-fee schools in Capricorn North and Capricorn South districts randomized into one of three arms. The three arms of the LP intervention (40 schools per arm) are a control group, a group that receive materials only (LTSM) and a group that receive materials and also have TAs in each class (LTSM+TA). The LTSM and LTSM+TA arms also receive training and are monitored by the programme team. There are around 2400 Grade 1 learners across the 120 schools.
In the baseline assessment, two different instruments were used to assess mathematics skills. The first was a one-on-one numeracy assessment the second was a written assessment administered to groups of up to 10 learners at a time. This instrument, a hybrid of the standard EGMA, was developed for Grade 1 learners and designed to be administered to groups of learners using tablets (Spaull et al, 2022). In the midline assessment, a collaboration with the developers of the MARKO D instrument was formed and a cross validation activity was carried out. Through this process the evaluation was validated against an internationally recognized instrument, strengthening the findings of the external evaluation while at the same time a Sepedi version of the MARKO D could be trialled and developed for use going forward.

This paper will present the methodology and the processes for developing, piloting, and validating the Sepedi version of the MARKO D instrument and the way in which the validation of the instruments used in the original design of the evaluation was carried out. Preliminary findings and lessons learned from utilizing the tools for data collection for the mid-line data collection for the project evaluation will be shared. This also contributes to the expansion of the body of evidence around what works to improve mathematics teaching and learning and will be used to open discussion on the potential for use and further development/adaptation of instruments that can be used for evaluation purposes and ways in which this can be done. The diagnostic potential of the MARKO D and ways in which such an instrument can be used will also be discussed (Fritz et al, 2020).

Author

• Ingrid Marta Sapire, School of Education, Wits University