Building and Assessing the Credibility of Causal Inference in Policy Evaluation

Session Submission Type: Panel

Abstract

As policy researchers face growing demand for credible evidence in real-world settings, they are often asked to make strong causal claims under conditions that complicate otherwise rigorous study designs. Administrative data may be incomplete or inconsistently defined, implementation may vary across sites, identifying assumptions may be difficult to assess directly, and policy environments may affect who can ethically or practically be included in research. Yet policy decisions still must be made, often on the basis of imperfect evidence. In this context, methodological discussion in applied policy research too often emphasizes estimator choice and statistical significance more than the broader question of credibility: why a particular study design and resulting evidence should be trusted for a particular policy question.

This panel brings together four papers that address this challenge. Collectively, they show that strengthening causal inference in policy evaluation requires explicit attention to study design, implementation context, data quality, subject selection, and the policy systems within which research takes place. The session highlights approaches for assessing, strengthening, and communicating causal credibility more systematically across randomized and nonrandomized policy evaluations.

The four papers approach this issue from different but complementary angles, pairing methodological innovation with real policy applications. Noor Qaragholi examines how qualitative inquiry can be used to assess identifying assumptions in causal analysis. Beth McGinty shows how mixed-methods evidence can strengthen target trial emulation by improving understanding of implementation and interpretation of findings. Elizabeth W. Eisenhauer compares quasi-experimental approaches using a Total Causal Error framework, emphasizing how measurement, representation, and identification issues shape the credibility of causal conclusions. Anna Marion examines policy interference in local government RCTs, highlighting trade-offs between participant protection and fair subject selection.

Together, these papers offer a broader view of rigor in policy evaluation: one that treats causal inference as a process of building, testing, and communicating credibility, not merely estimating effects. The panel will be of interest to policy researchers, evaluators, and practitioners seeking practical tools for assessing study designs, assumptions, implementation conditions, and the interpretation of evidence when no single methodological feature is sufficient to establish credibility. By focusing on how credibility is established in practice, the session speaks directly to APPAM’s applied audience and to the broader challenge of producing policy-relevant evidence that decision-makers can use with confidence.

Policy Area

• Methods and Tools of Analysis

Chair

• Noman Khanani, WestEd

Discussant

• Michael J Maranda, Retired

Organizer

• Elizabeth Willow Eisenhauer, Westat

Individual Presentations

• Mixed Methods for Improving Causal Inference in Policy Evaluation - Presenting Author: Noor Qaragholi, Johns Hopkins University

• Impact of integrated Medicare-Medicaid plans for dual eligibles with serious mental illness: a mixed-methods study - Presenting Author: Beth McGinty, Cornell University

• Assessing Quasi-Experimental Approaches for Evaluating a College Coaching Program for Students with Foster Care Experience - Presenting Author: Elizabeth Willow Eisenhauer, Westat; Non-Presenting Co-Author: Kevin P Baier, Westat; Non-Presenting Co-Author: John Cosgrove, Westat; Non-Presenting Co-Author: John D Fluke, University of Colorado Denver; Non-Presenting Co-Author: Daifeng Han, Westat; Non-Presenting Co-Author: Rachel Mayes Strawn, Virginia's Community Colleges; Non-Presenting Co-Author: Kathryn A Henderson, Westat

• Managing Policy Interference and Fair Subject Selection in Local Government RCTs - Presenting Author: Anna Marion, University of North Carolina at Chapel Hill