How Much Shall We Trust IV Estimates? Practical Advice Based on 63 Replications

• In Event: Observational and Experimental Causal Inference

Abstract

Instrumental variable (IV) strategies are commonly used in political science to establish causal relationships, yet the identifying assumptions required by an IV strategy are demanding, and it remains challenging for researchers to correctly apply this method. We replicate 63 papers published in three top journals in political science during the past decade and find several troubling patterns: (1) researchers often miscalculate the first-stage F statistics, thus overestimating the strength of their IVs; (2) most researchers rely on classical asymptotic standard errors, which often severely underestimate the uncertainties around the two-stage-least-squared (2SLS) estimates; (3) in the majority of the replicated studies, the 2SLS estimates are much bigger than the ordinary-least-squared estimates, and their ratio is negatively correlated with the strength of the IVs when the IVs are non-experimental, suggesting potential violation of the exclusion restriction; such a relationship does not exist when the IVs are produced by experiments. To improve practice, we provide a checklist for researchers to avoid these pitfalls and recommend a local-to-zero test to guard against failure of the identifying assumptions.

Authors

• Apoorva Lal, Stanford University

• Mackenzie Lockhart, UC San Diego

• Yiqing Xu, Stanford University

• Ziwen Zu