Off the Grid? The Impact of Questionnaire Design On Web Survey Response Quality

• In Event: Innovations in Survey Sampling and Questionnaire Design

Abstract

Web surveys have become ubiquitous in political science research, and while the use of attention checks have reduced satisficing and inattentive respondents, data quality concerns remain. Researchers often present multiple questions from a survey battery using a grid to save space, but studies suggest this may increase satisficing. I examine a number of question batteries borrowed from the ANES and CES that require close reading and solicit responses that should be internally valid. Respondents are randomly assigned to either complete the battery with a single question on every page, or in a single grid. I test the batteries for internal consistency, examine response times to identify rates of “too fast” responses and compare results across survey designs. Finally, I replicate a survey experiment and examine whether questionnaire design increases inattentiveness to the extent of lowering observed treatment effects later in the survey. The evidence suggests that grid-based survey batteries may lower respondent attention and encourage faster, more uniform, and less consistent responses. I conclude by discussing the study’s limitations and implications for future public opinion research.

Author

• Zachary Lorico Hertz, University of Chicago