Addressing Design Effects in Double List Experiments

• In Event: Political Methodology Virtual Poster Session

Abstract

Political scientists use list experiments to address sensitive attitudes and behaviors, yet a recent review suggests that the technique is generally underpowered to capture the extent of sensitivity bias in most applications. An under-explored suggestion to overcome this limitation is to implement double list experiments, a variant of the list experiment that presents subjects with two baseline lists and randomly assigns the sensitive item to appear in one of the lists. However, to yield valid prevalence estimates, the researcher must guarantee that both lists are comparable. Aside from recommendations on how to construct lists, the discipline currently lacks a standard to determine whether two lists are indeed comparable, which means the researcher cannot disentangle a true effect from a design effect. Leveraging the placement of the sensitive item, this paper proposes a statistical test to detect design effects in double list experiments based on the difference in correlations across different treatment regimes. I illustrate the properties of this test with a reanalysis of a study on support for anti-immigration organizations in California, along with simulations and an original validation study.

Author

• Gustavo Diaz, Tulane University