Interviewer Effects in Face-to-face Survey in Politically Sensitive Context

• In Event: New Insights on the Internet and Authoritarian Rule in China

Abstract

The organization of face-to-face political and social surveys is fraught with the risk that interviewers may involuntarily influence respondents and thus result in biased estimates. "Interviewer bias" is particularly problematic in the context of authoritarian systems where citizens--even when they are reassured about anonymity of the data collection as well as their right to opt out of questions they deem too sensitive--may nevertheless feel the need to "please" the interviewer by signaling support of official policies or loyalty to the regime. At the same time, researchers who organize face-to-face surveys typically need to dispatch interviewers to specific geographical zones for the sake of operational convenience and thus avoid sending the same team members to multiple localities. As a result, it is often difficult to disentangle neighborhood effects" from "interviewer effect" at the analysis stage.

In this paper, we examine how to disentangle these effects, using information from a unique nationally representative survey of China organized in 2018 in 75 localities in which interviews conducted face-to-face interviews or randomly selected adults. We take advantage of a unique feature of the research since each field interviewer was asked to answer the entire questionnaire during training "as if" they were respondents. This parallel dataset enables the researchers to control for item-specific interviewer effect at the analysis stage while at the same time accounting for local effects.

We demonstrate that even though interviewer effects are not ignorable, the magnitude of these effects vary with the placement of items in the questionnaire [early versus late], the cognitive difficulty of the item, as well the degree of political sensitivity of the issue. At the same time, locational effects [based on the specific geographical clustering of the respondents] are also substantively important. Thus, we recommend that the analysis of face-to-face date, especially in authoritarian contexts should make use of both sets of controls in order to reduce estimation bias.

Author

• Pierre F. Landry, Chinese University of Hong Kong