Rethinking Van Evera’s Tests in a Probabilistic Framework

• In Event: Process Tracing. New Directions in Qualitative Research

Abstract

In recent years, Van Evera’s process-tracing tests have been increasingly adopted as an inferential methodology for qualitative research, both in political science, and beyond the academy in the field of policy evaluation. Yet many treatments draw inappropriate conclusions about the inferential logic of Van Evera’s tests by incorrectly extrapolating from the “deductive limit," where evidentiary outcomes are strictly necessary and/or sufficient. Relatedly, many treatments fail to recognize the critical role that rival hypotheses play in determining both test type and test strength.

This paper critiques and builds on recent initiatives that have aimed to place Van Evera’s four test types within a fully probabilistic Bayesian framework by drawing on the concepts of sensitivity and specificity, which are commonly used in medical diagnostics and frequentist statistics literature. These approaches map test type onto a “probative value space” defined by the respective likelihoods of a binary clue (evidence that can be either present or absent) under two rival hypotheses (Bennett 2015, Humphreys & Jacobs 2015). However, Bayesian test strength depends not on likelihoods, but on likelihood ratios—the latter quantities are what govern updating. Moreover, logarithmic scales—which are standard practice in Bayesian analysis and information theory—are more appropriate than linear scales for measuring how much we learn from a given test. Accordingly, we present three improved approaches that classify test strength in terms of (1) weights of evidence (the logarithm of the likelihood ratio), (2) relative entropies, and (3) expected information gain. Weights of evidence describe post-data learning, while relative entropies and expected information gain may be of interest when making ex-ante decisions about what data to gather.

Ultimately, we emphasize that inference and test strength are always matters of degree, not type. But if a test typology is nevertheless desired, understanding the Bayesian foundations that we explicate can help practitioners avoid a number of common errors and better assess the strength of the tests that their evidence provides.

Authors

• Tasha A. Fairfield, London School of Economics

• Andrew Charman, University of California, Berkeley