The Evidence Paradox: When Education Systems Block the Holistic Evidence They Seek to Enable

• In Event: Rethinking Education Systems: Equity, Evidence, and the Challenge of Supporting Holistic Learning Worldwide

Proposal

Introduction

Evidence is a central concept in educational policy and practice, often used to justify decisions, guide implementation, evaluate interventions and steer funding. Yet, what counts as valid evidence remains contested across stakeholder groups (Kucirkova & Dockterman, 2024). Teachers may prioritise classroom observations, researchers emphasise empirical studies, and global agencies typically rely on standardised test scores. While efforts have been made to consolidate diverse perspectives into unified frameworks (see e.g. Baek et al., 2024), systemic barriers persist to advance a shared understanding of what constitutes valid evidence in education.

To explore these tensions, this study draws on an evaluation of the ENJOY Evidence Framework developed by the Jacobs Foundation (2022–2025). The ENJOY framework consolidates eleven leading evidence standards and operationalises them into twelve dimensions across four categories: evidence culture, effectiveness, implementability, and transferability. The framework is operationalized through the self-assessment ENJOY Navigator Tool, which enables organisations to self-assess their evidence practices and access tailored resources. The tool is a multidimensional evidence framework developed between 2022–2025 to support reflection on evidence quality. Together with the ENJOY framework’s comprehensiveness and recognition among educational institutions, the Navigator made it a suitable reference point for examining different perspectives on evidence among educational leaders.

This paper investigates how educational leaders perceive valid evidence, the challenges they face in applying it, and the role of self-assessment tools like ENJOY Navigator Tool in supporting evidence-informed practice. It introduces the concept of the evidence paradox, which draws on the empirical findings and theorizes evidence within the new dimensions although leaders aspire to a holistic understanding of evidence, systemic factors—such as political agendas and funding constraints—fragment this vision into narrow, context-specific definitions.

Methodology

This qualitative study involved remote interviews with fifteen educational leaders affiliated with the Jacobs Foundation. Participants were recruited via snowball sampling and held senior roles such as CEOs, Directors of Impact, and Heads of Evaluation. All participants had prior engagement with the ENJOY Navigator Tool (Jacobs Foundation, 2025).

Interviews were conducted via Zoom, transcribed, and analysed using a three-cycle coding process. The first cycle employed In Vivo coding to capture participants’ language. The second cycle used pattern coding to group initial codes into themes (Saldaña, 2013). The third cycle followed Clarke and Braun’s (2017) thematic analysis procedure, involving recursive phases of familiarisation, coding, theme development, and refinement. The ENJOY framework’s four dimensions, spanning evidence culture, effectiveness, implementability, and transferability, served as a reference point for structuring the interview protocol and interpreting responses.

Ethical safeguards included verbal and written consent, anonymisation of quotes, and assurance that data would not be shared with the Jacobs Foundation. The latter was a crucial safeguard to ensure that participants could speak freely without concern that their responses might influence future funding opportunities. All participants reviewed the final draft of the paper prior to journal submission.

Findings

Thematic analysis revealed three dominant perspectives on evidence: hierarchical, pluralistic, and multidimensional definitions of evidence. The hierarchical perspective prioritised RCTs as the gold standard as per e.g. the GRADE Working Group, 2004; Brożek et al., 2009), with participants acknowledging their methodological rigour but also limitations in capturing contextual variables and feasibility in real-world settings. The pluralistic perspective emphasised methodological fit, selecting evidence types based on context and stakeholder needs (Parkhurst & Abeysinghe, 2016). Leaders with this view noted the lack of standardised procedures for qualitative data and the need for protocols that value practitioner insights (Schlosser & Sigafoos, 2009).

The multidimensional perspective treated various dimensions, such as contextual relevance, stakeholder engagement, and practical applicability, as interdependent. Leaders adopting this view advocated for frameworks that integrate diverse evidence types but noted challenges in operationalising such approaches due to limited capacity and technical expertise (UNESCO, 2024; Kucirkova, 2023).

In addition to the key three definitions of evidence, a fourth theme, the governance perspective, highlighted how evidence is shaped by socio-political and funding agendas. Participants described how funders’ expectations often dictate evidence standards, leading to a narrow focus on measurable outcomes and discouraging learning from failure (Kelly & Pande, 2025; Mendizabal & Weyrauch, 2024). Within this theme, evidence was frequently framed as a tool for securing funding rather than for organisational learning.

The findings together with a comprehensive literature review and authors’ reflections, contributed to the formulation of the evidence paradox. The evidence paradox states that individual aspirations for holistic evidence are undermined by systemic constraints. While leaders of educational organisations with long-standing experience in working with evidence recognised its complexity (whether hierarchical, multidimensional, or pluralistic) and acknowledged the limitations of their own definitions, they converged on one point: the governance of evidence ultimately defines its use, shaped by what funders and systems demand. The paradox lies in the fact that although there is a shared desire to achieve a holistic definition of evidence to address systemic issues in education, governance mechanisms fragment evidence back into narrow categories aligned with funders’ priorities and institutional agendas.

Participants, especially those who referenced the ENJOY framework, found such frameworks and tools to be valuable in assessing evidence and fostering strategic thinking about the organizations processes. They all acknowledged that with multidimensional framework and tools such as ENJOY, come operationalization challenges, such as how to apply the insights into organizational processes efficiently and effectively.

Conclusion

The study concludes that addressing the evidence paradox requires repositioning education systems as learning systems, that is entities that adapt based on iterative learning and openly reflect on mistakes (Hailey & James, 2002; Odor, 2019). Such systems must be supported by funders and policymakers who embrace a learning orientation and promote flexible, context-sensitive approaches to evidence generation and use.

Authors

• Jimmie Odhiambo, University of Wisconsin Madison

• Natalia Kucirkova, University of Stavanger