Expert Surveys and Self-censorship in Authoritarian States

• In Event: New Methods and Measurement Strategies in the Study of Democracy & Autocracy

Abstract

Many political science datasets rely on expert ratings for latent attributes. While scholars are aware of variation in individual responses when aggregating such data, few studies have systematically examined the factors affecting coders' scale perception and levels of idiosyncratic error. The issues may be particularly severe when measuring sensitive concepts for authoritarian countries based on ratings provided by experts residing in those countries. Experts may self-censor themselves in expectation of government sanctions. To explore whether self-censorship in autocracies affects expert reliability and scale perception, we utilize coder-level data compiled by the Varieties of Democracy project, which measures various aspects of democracy cross-nationally and relies on both domestic and foreign experts. The data are aggregated with an ordinal item response theory model that parameterizes expert reliability and thresholds on the latent scale. We find that experts based at institutions in autocracies tend to be less reliable and have different thresholds compared to democracy-based experts for the same units. The tendency is particularly evident for variables related to autocrats’ commitment ability to media freedom.

Authors

• Hans Hanpu Tung, National Taiwan University

• Yi-ting Wang, National Cheng Kung University