Beyond evidence hierarchies: Uncertainty and its consequences in evidence-based decision making

• In Event: Evidence-based decision making in education
  In Formal Panel Session: Evidence-based decision making in education: asking the right questions, finding the right evidence

Proposal

Based on a biomedical model, policy-makers in education explicitly or implicitly view evidence in a hierarchy according to its potential to produced unbiased conclusions. A systematic review of randomized controlled trials is at the top of the hierarchy with case studies at the bottom. We argue that a more useful way to assess an evidence base is by considering the degree of certainty with which a policy decision can be made based on the evidence. We present an approach that can be taken to assess an evidence base along these lines. The first step is to map out the possible outcomes of a policy decision, including undesirable as well as the intended positive outcomes. The second step is to review the evidence on whether the policy action will lead to each of the identified outcomes. The evidence review should articulate – and where possible quantify - uncertainty in the relationship between action and outcome. Uncertainty can take several forms. The term is most commonly associated with statistical uncertainty of parameter estimates, typically expressed in confidence intervals. Other forms of uncertainty include methodological uncertainty arising from assumptions implicit in a research method and uncertainty in the applicability of research findings to a new context, time or program design. Policy decisions should be taken by considering the inherent uncertainty and its consequences. We seek greater certainty for severe negative consequences. Put simply, if severe negative consequences can be ruled out, we can tolerate greater uncertainty in positive outcomes. The application of this framework to decision-making helps us move beyond rigid hierarchies of evidence to assess the kinds of evidence required – and the degree of uncertainty we can accept – for different types of decisions. The approach can help decision-making based on partial evidence. The framework has implications for the generation of evidence too. Education policy evaluations should systematically consider potential negative outcomes. Investment in evaluations should be made in reducing uncertainty in outcomes with the biggest consequences. Uncertainty can be managed by placing small bets to achieve large goals. We use examples of policy decisions in school health and research on education systems to demonstrate that more systematic analysis of uncertainty and its consequences can improve approaches to decision-making and to the generation of evidence.

Authors

• Anne Buffardi, Overseas Development Institute

• Matthew Jukes, RTI International