Trust but Verify – Mechanistic Heterogeneity and Multi-method Research

• In Event: Advances in Mixed Methodology

Abstract

Multi-method research has gained considerable traction over the last two decades as a strategy to derive robust causal inferences in the social sciences (Lieberman 2005; Goertz 2017; Humphrey & Jacobs 2015; Seawright 2016; Weller & Barnes 2014). In a nutshell, the basic ideas behind many recent multi-method proposals is that in-depth case studies are utilized to make inferences about causal mechanisms, combined with large/medium-n cross-case comparisons that are used to find regular associations across cases, to select appropriate cases that enable generalizations to be made based on findings from case studies. In and of themselves, in-depth case studies using process-tracing only enable inferences about causal mechanisms to be made about the studied case. This means that unless we want to engage in a mission impossible of studying all cases in a population, we need to use data from the cross-case comparisons to generalize beyond the studied case.

Unfortunately, most of the existing literature has focused on the establishing cross-case similarity as the main criterion for generalization, thereby assuming mechanistic homogeneity. We define mechanistic heterogeneity as a situation where the same set of conditions is linked to the same outcome through different causal mechanisms in two or more cases.

We contend that the contextual sensitivity of mechanistic explanations - in particular when unpacked as systems - means that mechanistic heterogeneity might be lurking under what might looks like a homogeneous set of cases at the level of causes. However, a look at the methodological guidelines on multimethod research and case selection strategies shows that neither variance-based (Gerring 2017; Nielsen 2016; Seawright 2016; Weller and Barnes 2014) nor case-based approaches (Beach and Rohlfing 2018; Goertz 2017; Rohlfing and Schneider 2018; Schneider and Rohlfing 2013, 2016) properly take into account mechanistic heterogeneity, but instead resolve the issue merely assuming that cross-case causal homogeneity implies mechanistic homogeneity.

The paper starts by providing a simple example of mechanistic heterogeneity, in which we show in a published article that used QCA that different mechanisms are actually lurking underneath a set of cases that the QCA analysis told us were causally homogenous.

The article then reviews the existing variance- and case-based literature for case selection and generalization, showing that it is blind the problem of mechanistic heterogeneity lurking underneath a set of cases that our cross-case analysis told us were similar. We find that when using existing standards, there is a large risk of making flawed mechanistic generalizations. In this review, we delineate five different sources of potential mechanistic heterogeneity, for example showing how cross-case QCA or regression analysis might lead to us to conclude that a given condition does not produce a difference in the outcome. However, the cross-case analysis provides us with no information about whether the condition matters for what mechanisms are at play.

The final section introduces a snowballing-outwards strategy for generalizing about mechanisms within a bounded population of cases that can reduce the risk of flawed generalizations about mechanisms. While the risk of mechanistic heterogeneity produced by the contextual sensitivity of mechanisms naturally reduces the scope of valid generalizations about mechanisms, we contend that the goal of cumulative research about mechanisms should be to delineate the proper bounds within which specific mechanisms work, enabling us to more confidently claim that the mechanism will work as hypothesized in a particular context.

Authors

• Derek Beach, University of Aarhus

• Markus Bjoern Siewert, TU Munich