Narratives of Response Error from Cognitive Interviews of Survey Questions

• In Event: 576 - Section on Methodology Paper Session. Integrating Multiple Methodologies into Data Collection and Analysis

Abstract

That rates of normative behaviors produced by sample surveys are higher than actual behavior warrants is well evidenced in the research literature. Less well understood is the source of this error. While recent research has argued for a cause based in the respondent’s self-concept, this explanation has been tested primarily in correlational models. Thus, to investigate the source of bias in survey estimates of one such normative behavior, religious service attendance, 25 cognitive interviews were conducted to probe responses to a set of common, conventional survey questions about attendance and other religious behaviors. Respondents were asked to retrospectively think-aloud, describing how they arrived at the answer they reported. Answers to these conventional survey questions and cognitive probes are compared both quantitatively and qualitatively. Half of the respondents amended their answer during cognitive probing, all explicitly or implicitly indicating a lower rate of attendance than originally reported, yielding a statistically significant reduction in reported attendance. Narrative responses shed light onto the source of measurement bias as respondents pragmatically interpreted the survey question to allow themselves to include other types of religious behavior, to report on a more distant and more religious past, and discount current constraints on their religious behavior, in order to allow reported behavior to match aspirational or normative religious identities.

Author

• Philip S. Brenner, University of Massachusetts-Boston