Observational Open Science: Application to the Literature on Irrelevant Events

• In Event: Causal Inference: Design and Generalization

Abstract

The credibility of scientific inquiry depends on researchers’ ability to replicate findings from prior work. However, publication biases that favor statistically significant results reward research practices that undermine replicability and increase the probability of false positives, including specification search and selective reporting. In the wake of recent failures to verify published research findings, research communities are increasingly adopting transparent and reproducible research paradigms.

Extant frameworks to address the “replication crisis” highlight the importance of large-scale replications of experiments (Open Science Collaboration 2015) and pre-registered analysis plans (PAP) (Casey et al. 2012). This focus is not surprising given the contemporary “open science”' movement’s focus on randomized experiments. However, many important social science findings, including the majority of political science research, come from analyses of observational data---data the world has already produced---like election outcomes, incidents of war, and legislative votes.

Lacking access to new data, the toolkit for probing the robustness of observational findings often amounts to (1) making modifications to the original empirical strategy and/or (2) introducing new cases or topics in which a similar pattern should emerge. These practices have their place. However, as strategies for assessing a past finding’s validity, both suffer from a core pathology that replication is meant to prevent. In particular, given a wide range of possible alternative cases and specifications, researchers may be able to “null hack” findings away. Current practices for replicating observational research often produce contentious and confusing exchanges about optimal specifications, data, and research designs. Too frequently, the legitimate exercise of challenging an original study’s methods gets bogged down by the possibility that both sides engaged in selective reporting, leaving consumers of research in the unenviable position of trying to adjudicate between studies without complete visibility into the process that led to each study’s methodological choices.

In light of these issues, this paper describes an effort to improve the toolkit for evaluating the reliability of results from observational studies. We refer to this as an observational open science (OOS) approach.

Our approach comprises five features:

First, we leverage the passage of time to extend the time series without changing the original study’s units of analysis. Second, we collect all data independently instead of simply using replication data from the original studies. Third, as with experimental open science, we pre-register all analyses. Fourth, we conduct multiple simultaneous replications of separate studies testing the same theoretical claim. Fifth, this study represents a collaboration between authors with mixed incentives. For both observational and experimental replication studies, collaborations that involve mixed incentives---some authors with a stake in the original results and others with no such stake (and potentially a motivation to overturn the original findings)---make replication efforts more credible.

We apply this OOS approach to the literature on irrelevant events and voting behavior. A core question in political science concerns democratic accountability, or the extent to which elections are effective tools for enhancing social welfare via the electoral sanctioning and selection of leaders. Recent work challenges canonical models of retrospective voting by arguing that voters often reward and punish incumbents for shifts in affect caused by events beyond their control (e.g., Achen and Bartels 2016). Other studies are skeptical of these claims (e.g., Fowler and Hall 2018).

An open science approach can help clarify these muddy waters, not only for the literature on irrelevant events, but also for other areas with mixed and inconclusive findings. We apply our OOS approach to three prominent studies in the irrelevant events literature: (1) Achen and Bartels’ (2016) well-known finding that incumbents are electorally harmed by abnormal precipitation (droughts and floods); (2) Healy et al.’s (2010) finding that the results of college football games immediately preceding Election Day influence election results; and (3) Healy and Malhotra’s (2010) finding that incumbents are harmed by tornado damage but not by tornado deaths. Each study replicates well in some areas and poorly in others. Had we sought to debunk any of the three with ex post specification search, we could have done so. However, our approach required us to see the full, complicated picture. We conclude with suggestions for future refinements to our approach.

Authors

• Matthew H Graham, George Washington University

• Gregory Huber, Yale University

• Neil Malhotra, Stanford University

• Cecilia Hyunjung Mo, UC Berkeley