BackHome
Refresh: OffView Personal Schedule
Session Submission Summary

Designing for evidence-informed digital personalised learning: facilitating critical comparisons, understanding limitations, and establishing future directions

Tue, March 25, 4:30 to 5:45pm, Palmer House, Floor: 7th Floor, Clark 10

Group Submission Type: Formal Panel Session

Proposal

This panel will critically examine how education stakeholders might more effectively make evidence-informed decisions about digital personalised learning (DPL), with a particular focus on implementing DPL in low- and middle-income countries (LMICs). Specifically, this session will consider how to address the limited consensus on the definitions and components of DPL, the challenges in comparing different approaches, and the limitations in assessing impact.

CONTEXT AND RELEVANCE:

An increasing number of tools, broadly classified under the umbrella term of DPL, are being adopted in education systems globally (UNICEF, 2022). However, there is a lack of consensus across the education sector about how to define and/or categorise these. Terms including computer-assisted learning, computer-aided learning, computer-aided instruction, personal adaptive learning, personalised learning environments, and intelligent/cognitive tutoring systems are amongst those that have been used interchangeably to describe interventions that personalise learning ([AUTHOR], 2020). Several recent frameworks have also been developed, each offering different approaches for understanding personalisation (e.g., Holmes et al., 2018; Van Schoors et al., 2021; Walkington & Bernacki, 2020).

This limited consensus has implications. First, the diversity of DPL approaches may explain the range of reported impacts on learning observed in the literature (de Barros & Ganimian, 2024). However, the lack of a commonly agreed definition of DPL, including its scope and limits, makes it challenging to compare findings from different studies, even between different tools used in the same educational context. Second, DPL tools themselves are multi-layered constructs, encompassing a range of diverse approaches – especially due to the complexity of the algorithm ‘black box’ that may be inherent in such tools (Bearman & Ajjawi, 2023). As such, there have been calls for DPL research to better define the mechanisms of personalisation adopted by each tool, and to identify key implementation factors (Van Schoors, Elen, Raes & Depaepe, 2023).

SESSION OBJECTIVES AND STRUCTURE:

This panel will involve three presentations and an extended discussion, all of which will centre around the question: how might we conceptualise and research DPL in a way which enables a critical comparison between different tools?

The panel will comprise researchers from [ANON] – a global research partnership which aims to empower people by giving them the robust evidence they need to make decisions about technology in education – as well as fellow researchers in the field providing critical reflections. After an introductory discussion to frame key concepts and objectives, the first two papers will present findings and critical reflections from two DPL research studies in Kenya and Sierra Leone. The third will present an amended framework for assessing the cost-effectiveness of DPL tools. In summary:

Paper 1 presents findings from a randomised controlled trial involving 291 schools in Kenya, assessing the impact of a DPL tool which is closely aligned with the pre-primary national curriculum and pedagogy. Through a critical discussion of the results, the importance of viewing DPL as a multi-layered construct when interpreting findings - and the potential implications if this for integrating DPL into education systems - are explored.

Paper 2 presents findings from a quasi-experimental, difference-in-difference study, assessing the impact of different implementation modalities of a DPL tool for lower-primary learners in Sierra Leone. The results, indicating that personalised and non-personalised modalities had similarly significant impact on learning, interrogate the extent to which impact can be attributed to the personalised features of DPL tools.

Paper 3 presents a cost-effectiveness analysis of the two studies outlined in the first two papers, with a focus on the challenges and limitations of comparing different DPL tools. The findings from these two studies are also situated within the context of other DPL interventions and the cost- effectiveness data used to represent them. Finally, an amended framework is proposed for more nuanced cost-effectiveness analysis, based on enhanced data on learning trajectories, learner characteristics and integration with curricular objectives.

Structured to allow opportunities for interaction, an online platform will capture audience questions and reflections, to facilitate exchange of ideas during and following the panel. The session will feature:

> An overview of the session framing, objectives and format by the Chair (10 mins)
> Paper presentations, followed by small-group discussion (10 + 5 mins, x3)
> Discussant and panel reflections (5 mins)
> Audience-wide discussion and close (15 mins)

CONTRIBUTION AND SIGNIFICANCE:

Educational technologies, including those that personalise aspects of learning, have repeatedly been touted as a way to significantly improve learning; however, too often there has been a lack of real-world impact and cumulative progress (An & Oliver, 2021; [AUTHOR], 2024). This session seeks to facilitate a critical discussion about the promise of DPL to enhance educational outcomes. Drawing on results from two empirical studies and a cost-effectiveness analysis conducted in sub-Saharan Africa, it will interrogate the extent to which these findings are generalisable and consider the contribution of the studies to the wider sector. Crucially, the panel will disentangle the ways in which DPL is currently conceptualised and consider the implications of this for research and practice.

In addition to the individual presentations, the panel will encourage wider discussions about making evidence-informed decisions on DPL. It will facilitate the sharing of insights from across the sector, including the identification of common challenges, and a particular consideration of issues related to integrating DPL in LMICs. The goal of the session is to advance the conversation beyond an understanding of DPL’s potential impact on foundational learning outcomes, to a more nuanced discussion about the design features, implementation models, and sustainability factors that make assessing impact challenging, but also present opportunities for meaningful decision-making.

Sub Unit

Chair

Individual Presentations

Discussant