Concrete Counterfactual Tests for Process-tracing

• In Event: Process Tracing. New Directions in Qualitative Research

Abstract

There exists a variety of approaches for making causal inferences in process-tracing, each based on a different philosophical conception of what a causal mechanism is (George and Bennett 2005; Jacobs 2016; Mahoney 2001). These different approaches give conflicting methodological advice and so it is crucial to commit to one of them.

In this paper, I defend a variance-based (‘interventionist’) approach, based on the work of philosopher of causation James Woodward (Runhardt 2014, 2016; Woodward 2003). This approach is based on counterfactual analysis and as such is often criticized for purportedly being unable to provide concrete evidential tests (Beach 2016). In the first part of the paper, I meet this criticism head-on by presenting a new, concrete, counterfactual-based evidential framework. In particular, I defend Woodward’s approach to causal inferences as fruitful because it implies particular, relatively unexplored tests for the strength and quality of process-tracing evidence. I end the first part of the paper by specifying what such tests would look like for political science.

In the second part of the paper, I illustrate the framework with a concrete example, Haggard and Kaufman’s process-tracing work in Dictators and Democrats (Haggard and Kaufman 2016). Evaluating Haggard and Kaufman’s work by causal interventionist standards, this paper finds that the authors do not meet interventonism’s strict demands for corroborating singular causal claims. Although the authors circumscribe the singular causal claim being tested in line with interventionist demands, they do not deliver sufficient evidence for the associated counterfactual. However, the authors are able to successfully reject alternative causal hypotheses for the singular case studies under investigation.

In the last part of the paper, I frame what came before in Bayesian terms. I argue my evidential tests for process-tracing are ‘hoop tests’ (Van Evera 1997): while we can be concrete about what the ‘counterfactual’ evidential tests ask of a researcher, such tests will be most powerful at eliminating rival causal inferences. The tests do not by themselves affirm the causal inference hypothesized (Collier 2011). In Haggard and Kaufman’s case, this means that while they have arguably successfully eliminated the distributive conflict hypotheses for particular Third Wave transitions, they have not yet affirmed their own hypotheses on the role of institutional strength and capacity for collective action in these Third Wave case studies.

References

Beach, Derek. 2016. 'What Are We Actually Tracing? Process Tracing and the Benefits of Conceptualizing Causal Mechanisms as Systems', Qualitative & Multi-method Research: 15-22.
Collier, David. 2011. 'Understanding Process Tracing', PS: Political Science and Politics, 44: 823-30.
George, Alexander L., and Andrew Bennett. 2005. Case Studies and Theory Development in the Social Sciences (MIT Press: Cambridge).
Haggard, Stephan, and Robert R. Kaufman. 2016. Dictators and democrats : masses, elites, and regime change (Princeton : Princeton University Press).
Jacobs, Alan M. 2016. 'Introduction: Mechanisms and Process Tracing', Qualitative & Multi-method Research: 13-15.
Mahoney, James. 2001. 'Beyond Correlational Analysis: Recent Innovations in Theory and Method', Sociological Forum, 16: 575-93.
Runhardt, Rosa. 2014. 'Evidence for Causal Mechanisms in Social Science: Recommendations from Woodward’s Manipulability Theory of Causation', Philosophy of Science, 82.
———. 2016. 'Tracing the Productive Continuity of Social Mechanisms', Qualitative & Multi-method Research, 14: 22-28.
Van Evera, Stephen. 1997. Guide to methods for students of political science (Ithaca: Cornell University Press).
Woodward, James. 2003. Making Things Happen: a Theory of Causal Explanation (Oxford University Press: Oxford).

Author

• Rosa Willemijn Runhardt, Leiden University