Field Evidence for Enumerator-Induced Treatment Heterogeneity in Experiments

• In Event: The Methodology of Experiments

Abstract

Existing work on enumerator effects has focused largely on survey contexts, in which enumerators may activate social desirability bias or affect non-response rates. In experimental settings, however, enumerators do far more than administer surveys; they lead discussion groups, implement complex experimental protocols, and deliver key pieces of information to subjects. Despite the centrality of enumerators in experimental protocols, however, researchers typically pay little attention to the consequences of enumerator-induced variation, believing that random assignment of enumerators eliminates threats to inference that might arise from them. In this article, we evaluate the inferential consequences and substantive magnitude of enumerator-induced treatment effect heterogeneity and provide methods to address the problem. First, we formalize the estimation of treatment effects under enumerator-induced heterogeneity and show that, contrary to conventional wisdom, enumerator effects have inferential implications under nearly all of the most common experimental designs and sampling schemes. Second, we use a combination of original and publicly available data from 12 experiments---including survey, lab-in-the-field and large-scale field interventions---to provide what is to our knowledge the first systematic evidence of the size and distribution of enumerator effects in experimental studies. We find evidence that enumerator effects are large in substantive terms, non-normally distributed, and can be predicted by enumerators' psychological and demographic characteristics. We also show that the common practice of including enumerator fixed-effects can produce substantial bias in the estimates of the average treatment effect. Finally, we propose a split-pot design and accompanying treatment assignment algorithm that can significantly reduce the inferential threat posed by enumerator effects. We show that this algorithm, which we have implemented in several studies, improves precision, reduces bias, and can achieve balance along other dimensions along which researchers often stratify treatment.

Authors

• Brandon de la Cuesta, Princeton University

• Lucy E. S. Martin, University of North Carolina, Chapel Hill

• Jing Qian, Princeton University