Mixed Methods for Improving Causal Inference in Policy Evaluation

• In Event: Building and Assessing the Credibility of Causal Inference in Policy Evaluation

Abstract

Introduction: Causal evidence plays a central role in decision-making across policy areas. The challenge is that estimating causal effects almost always rests on strong, untestable assumptions whose plausibility depends on the specific study context. Assessing that plausibility requires substantive knowledge that quantitative methods alone cannot provide. When assumptions are violated, policy evaluations can produce biased estimates and misleading conclusions, with important implications for funding, scale-up, and implementation decisions. Despite widespread acknowledgment of the importance of substantive knowledge for credible causal inference, structured approaches for generating and incorporating that knowledge into causal analyses remain limited.

Purpose: This paper explores how a mixed methods approach can systematically integrate qualitative inquiry with quantitative causal analyses to strengthen study design and assess the plausibility of identifying assumptions within a given study context. We demonstrate this approach using the complier average causal effect (CACE), which estimates the effect of an intervention among those who would participate as intended, and focus on two commonly invoked assumptions: principal ignorability (via principal score approach) and the exclusion restriction (via instrumental variable approach).

Data: Quantitative data are drawn from the Youth Empowerment Study (YES), a randomized trial of a trauma-informed intervention for youth involved in the juvenile legal system (n = 630) across 19 sites in New Mexico and West Virginia. Qualitative data comprise 10 semi-structured interviews with key informants purposively selected based on depth of experience with the intervention and its implementation.

Design/Methods: We estimate the CACE for two outcomes, self-regulation and recent vaginal sex, using both instrumental variable and principal score approaches. We analyze the 10 interviews using inductive thematic analysis followed by deductive mapping to the Capability, Opportunity, Motivation-Behavior (COM-B) model and the Theoretical Domains Framework (TDF). We synthesize these findings into the CACE-MM framework, a novel mixed methods approach for integrating qualitative inquiry with CACE analyses. The framework specifies how to (1) design qualitative inquiry to probe identifying assumptions, (2) systematically link qualitative evidence to those assumptions, and (3) incorporate this evidence into the interpretation of causal estimates and the assessment of their credibility.

Results: For self-regulation, both approaches yielded similar estimates, suggesting a small, favorable but nonsignificant effect among compliers. For recent vaginal sex, estimates were also broadly consistent, suggesting a modest reduction. Careful examination of the identifying assumptions, however, raised concerns about both approaches. The qualitative inquiry identified numerous constructs, including trauma exposure, that plausibly predict both participation and outcomes but were not measured, raising concerns about principal ignorability. For the exclusion restriction, qualitative analysis revealed meaningful intervention components not previously identified, and found that no defensible participation threshold could be identified for recent vaginal sex.

Conclusion: The CACE-MM framework provides policy researchers with a structured approach for incorporating qualitative evidence to strengthen study design and assess identifying assumptions. Although developed in the context of CACE, the approach extends to other causal topics that require substantive knowledge. A mixed methods approach supports more credible causal findings in applied policy settings by making the consideration of underlying assumptions more transparent and empirically grounded.

Author

• Noor Qaragholi, Johns Hopkins University
  Presenting Author