What policymakers want – documenting a mismatch in supply and demand of policy evidence

• In Event: Information, Communication, and Citizen-Government Encounters

Abstract

Policymakers are facing growing pressure to incorporate evidence into decision-making - but are they able to find, translate, and use the evidence they need? Recent findings suggest that the adoption of evidence-based policies remains low – less than 30% of nudge interventions for government mailers tested by a national nudge unit were ultimately adopted by the partnering cities (DellaVigna et al., 2024).

In this research, we build on a growing literature in public management and economics that examines barriers to evidence adoption in government, including information frictions (e.g., Hjort et al., 2021), policymakers’ preferences and biases (e.g., Baekgaard et al., 2019; Head, 2015; Vivalt & Coville, 2023; Xu et al., 2024), as well as organizational cultures and capacity (e.g., Crowley et al., 2021; Moynihan & Pandey, 2010). We investigate which evidence dimensions (e.g. effect sizes, evaluation methods, costs, implementation considerations, etc.) policymakers require to determine “what works” – and whether their demand is met by policy research.

In online experiments with more than 2,000 local US policymakers, we test which evidence dimensions they value and demand across two experimental paradigms: a conjoint task, in which participants are asked to recommend one of two policy programs described by their evidence dimensions; and a mouse-click task in which policymakers can choose to view specific evidence dimensions before judging a program’s credibility and implementation feasibility.

We find consistent evidence that policymakers demand and value effect sizes above any other evidence dimension. For example, policymakers are 28pp more likely to recommend programs with large effects than with null effects – more than three times the impact of any other attribute we test. We also find that policymakers are relatively insensitive to evidence dimensions relevant to an evaluation’s internal validity (e.g. RCTs vs. focus groups) across paradigms. On the other hand, contextual factors - such as implementation considerations, political context, as well as costs – meaningfully shape policymakers’ decisions. Understanding what policymakers want and how they trade-off different evidence dimensions is a crucial first step in explaining low adoption of evidence.

We contrast our findings with an LLM-assisted landscape analysis of around 16,000 policy research documents, coded for the evidence dimensions they report. We find a clear mismatch between policymakers and researchers. Policymakers value several dimensions that are rarely reported in research documents. For example, political context – one of the most influential factors in our experiments – is mentioned in fewer than 12% of documents. At the same time, policymakers place relatively little weight on dimensions that researchers emphasize heavily: evaluation methods are reported in roughly 88% of documents.

Our evidence suggests that there is a systematic misalignment between what policymakers value and how evidence is commonly communicated. Closing this gap will require a bi-directional approach: training policymakers how to evaluate evidence (e.g. Mehmood et al., 2023) and requiring researchers to adapt how they produce and communicate evidence to meet policymakers’ needs (e.g. Toma & Bell, 2024). Future work should test interventions that align the supply and demand of evidence dimensions – bringing the promise of evidence-based policymaking closer to practice.

Author

• Max Spohn, Harvard University
  Presenting Author