A Practical Guide to Dealing with Attrition in Political Science Experiments

• In Event: Advances in Experimental Design

Abstract

While a cottage industry has developed around improving other areas of experimental design, the issue of attrition -- missingness on the outcome variable -- has often been ignored by practitioners; indeed, researchers have few tools -- and no easily accessible overview -- to pre-empt, detect and address attrition in experiments. Important advances in the study of attrition nearly exclusively focus on ex-post solutions (e.g. double-sampling/extreme value bounds), which, though valuable, do not help with the issue of easily identifying attrition that results in threats to inference or preventing its occurrence in the design stage. Our practical guide to assessing the type, scope and impact of attrition in experiments -- paired with an open-source R package -- provides a method for visualizing attrition across treatment conditions and over-time, as well as utilizing and comparing balance tests at precise moments in the study and, for studies suffering from problematic attrition, incorporating estimation and visualization of Manski bounds.

Authors

• Adeline Lo, University of Wisconsin, Madison

• Jonathan Renshon, University of Wisconsin, Madison