Individual Submission Summary
Share...

Direct link:

Addressing Non-sincere Responses in Rank-Order Survey Questions

Sun, September 3, 10:00 to 11:30am PDT (10:00 to 11:30am PDT), LACC, 303A

Abstract

We address a hidden bias in the analysis of rank-order survey questions in social sciences: the bias caused by "non-sincere" respondents who randomly rank available items instead of revealing their sincere preferences. Using survey experiments and simulations, we show that randomizing item order mitigates the impact of non-sincere ranking patterns when they are uniformly distributed. However, a problem remains when non-sincere respondents choose specific ranking profiles more often than others for reasons unrelated to the choice set, preventing the identification of quantities of interest based on resulting ranking data. Our proposed solution only requires simple, auxiliary rank-order questions, which estimate the proportion and the distribution of non-sincere and non-uniform ranking profiles. With original and replication studies, we show how addressing the hidden bias may change our understanding of public opinion on various political topics, including legislative representation, racial identity, the Black Lives Matter movement, and trust in elections.

Authors