Assessing different implementation modalities of a digital personalised learning tool for lower-primary learners: a difference-in-difference study in Sierra Leone

• In Event: Designing for evidence-informed digital personalised learning: facilitating critical comparisons, understanding limitations, and establishing future directions

Proposal

CONTEXT AND RELEVANCE:

A recent systematic review on digital personalised learning (DPL) revealed a diverse range of conceptualization and implementation of DPL in literature, and the difficulties in comparing their impact on learning (Van Schoors, 2021). Most of the reviewed literature were quantitative experimental studies comparing treatment (DPL) with a single control group. Such a focus on the digital versus non-digital divide, although valuable in determining the impact on learning, sometimes fails to capture the complexity of different implementation modalities. Therefore, further investigation into implementation modalities, specifically regarding the device-learner ratio, is needed to understand what is the most (cost-)effective modality in DPL interventions (Plaut, 2024).

RESEARCH FOCUS AND DESIGN:

The study addresses this, by comparing two different DPL modalities (with different device-learner ratios) with a non-DPL modality (projector) and a control group. The research assesses the impact of a DPL tool – [ANON]’s [ANON] content, delivered via [ANON] and implemented by [ANON] – used by lower-primary learners in Pujehun District, Sierra Leone. [ANON] offers both a child mode and a teacher mode. The former involves personalisation based on an on-the-day diagnostic test, providing literacy and numeracy content in English that match the child’s current level during learning sessions. The teacher mode is not personalised: designed for whole-class delivery via projectors, it allows teachers to select learning content for lessons.

The study comprised a quasi-experimental, difference-in-difference design, involving 32 randomly-selected public primary schools in Pujehun, one of the most deprived districts in Sierra Leone. The sample of 212 learners (Class 1-3) was split across four modalities: one control, one “standard” child mode (one tablet per learner), one child mode with tablet sharing between two learners, and one teacher mode via a projector. Numeracy and literacy were assessed at baseline (Nov-Dec 2023) and endline (July 2024) using Early Grade Reading Assessment and Early Grade Mathematics Assessment tools. Additional qualitative data, including surveys, interviews, classroom observations, school visit memos, and implementation journals were collected in April 2024 (midline) and July 2024 (endline).

FINDINGS:

Findings indicate that all three implementation modalities (standard, tablet-sharing and projector) significantly improved both literacy and numeracy, to the extent that Class 1 learners from all intervention arms outperformed Class 3 learners from control schools at endline. Significant impact on equity was observed: bottom performing students improved more than their high performing peers (especially in literacy) when using the standard modality; and girls and boys performed equally well with the intervention, whereas boys outperformed girls in control schools. However, no significant difference was observed between the effectiveness of the two DPL modalities, and the whole class approach.

CONTRIBUTION AND SIGNIFICANCE:

This paper offers a critical discussion of impact measurement regarding different DPL implementation modalities. The results support existing literature, suggesting DPL can positively impact numeracy and literacy (Major, Francis & Tsapali, 2021), while demanding a more nuanced evaluation of what aspects of the personalised features of DPL tools contribute to such an impact, and calling for more consideration of the contextual factors during implementation.

Author

• Annette Zhao, EdTech Hub