Process Tracing as a Small n Method for Evaluating the Impact of Outcomes-Based Contracts

• In Event: Threats to Validity in Impact Evaluation Research

Abstract

To date, evaluations of outcomes-based contracts, including Social Impact Bonds (SIBs), have often sought to deploy counterfactual evaluation designs: experiments and quasi-experiments. Designing such evaluations is often difficult, however, due to the absence of an obvious counterfactual group, a relatively small number of cases, or an intervention evolving over time. To address these challenges, there is growing interest among impact evaluators in a range of ‘small n’ impact evaluation methodologies. Unlike counterfactual evaluations which seek to attribute an intervention’s causal effect, ‘small n’ approaches seek to establish an intervention’s contribution to observable outcomes while accounting for other causes and contextual factors.

One such evaluation approach is Process Tracing, which uses generative causation to investigate the mechanisms linking cause and effect. In this paper, the analysis applies Process Tracing to evaluate a housing-first SIB in the UK called the Greater Manchester Homes Partnership (GM Homes), asking: In what ways, and through which mechanisms, did GM Homes produce systems-level effects? To answer these questions, the analysis tests three main hypotheses related to the program’s focus on asset-based working, innovation, and collaboration. Evidence for testing these hypotheses is drawn from 21 interviews, primary documents, user data, and secondary literature.

Methodologically, the study’s use of process tracing enabled the analysis to account for the complexity of the GM Homes program, which evolved over time and was delivered in a system where many factors contributed to outcomes that were achieved. In doing so, this study demonstrates how process tracing may be a useful approach for evaluating future outcome-based contracts - or other innovative policy tools - by offering new evidence-based insights without relying on large sample sizes, counterfactuals, or statistical controls.

Author

• Hilary Olson, University of Southern California
  Presenting Author