Using evidence standards to move beyond a focus on evidence on effectiveness to assess evidence on implementability and transferability

• In Event: ENJOY: A digital framework for assessing and promoting education practitioners’ evidence practices

Proposal

Problem
Building a common understanding of what “good evidence” looks like is important for organizations seeking to engage in or support evidence-informed decision making. However, there are many differing views on what constitutes “evidence,” what dimensions of evidence must be considered when making decisions, how to balance characteristics such as rigor and contextual relevance, and how to combine and value different types and levels of evidence in decisions.

Solution
To help address these issues and others, the author (an advisory organization that works with partners to make sense of and act on evidence in education) developed a set of evidence standards designed to help organizations establish a shared view of “good evidence” and make evidence-informed decisions about strategy, programming, and funding. The proposed standards are informed by a model of evidence-informed practice, based on the premise that optimal decision-making relies on the integration of current best evidence, individual preferences and actions, and the wider context and circumstances (e.g., political and organizational factors). The standards evaluate the quality of evidence on a given intervention on the following domains:

1. Effectiveness – the degree to which an intervention can deliver the intended outcome.
2. Implementability – the degree of ease with which an intervention can be implemented successfully in a given context.
3. Transferability and scalability – the ability to transfer an intervention to a new setting (balancing fidelity with adaptation) and the ability to expand the reach of interventions.

The evidence standards define each domain, propose domain-specific assessment criteria, and set guidelines for rating interventions (from low to high) on each domain. The evidence standards cover all three domains because all three are considered integral to achieving programmatic impact at scale. The standards also recognize that different types of evidence are relevant to for decision making. For instance, a funder may support investing in interventions that have evidence to suggest that they are implementable and have the potential for scale but are still nascent in terms of their evidence on effectiveness. Underlying the standards is a desire to promote early consideration of all three domains, and to promote ways to strengthen the evidence relating to these domains.

Approach
The author generated the evidence standards by considering existing, best practice evidence frameworks and own experience designing, testing and refining similar frameworks for practice. They also conducted consultations with the initial, intended user of the framework (a foundation in the child development and learning space) to ensure the standards reflected their values and would support potential use cases (e.g., supporting partners in their evidence journeys).

The Way Forward
The evidence standards are theoretical, focused on assessing the evidence base of education interventions rather than organizations, and reflect the values and priorities of the author's organization and its funder. More work is needed to test the standards’ practical applicability, considering whether and how the evidence standards might be used for different types of organizational decisions (particularly decisions that are not focused on interventions), and examining their applicability to a broad set of organizations.

Author

• Maryanna Abdo