Who Rates? Identity, Bias, and Democracy Ratings

• In Event: Conceptualizing and Measuring Democracy and Democratic Backsliding

Abstract

Rankings and ratings of states are influential tools of global governance, not to mention valuable sources of data for research. The growth of these benchmarks in recent years has been remarkable. Existing research has focused on demonstrating GPIs’ direct and indirect effects, including on states, market actors, and international organizations. In this paper, we focus on a prior question: Who gets to rate countries? In particular, how might the rater’s identity affect how countries are evaluated? These questions speak directly to the reliability and validity of rankings and ratings of states, as well as to normative debates about who counts as an expert in world politics.

This paper takes up these questions through an in-depth study of democracy assessments. Specifically, we conduct two expert surveys in order to a) measure variation in democracy ratings across country experts, including academics, who are nationals of the country being evaluated and those who are not, and b) measure the relative authority of ratings conducted by experts who are and are not nationals of the country being evaluated. We focus on the case of Uganda, a hybrid regime that holds multiparty elections but has not had a change of power at the level of executive for over 30 years.

Ugandan and non-Ugandan (i.e., “international”) country experts in our study rated how democratic Uganda is using a methodology similar to that used by the organization Freedom House. We find that Ugandan experts rate Uganda as significantly and substantively “freer” than international experts. Some of this difference seems to be linked to conceptualizations of democracy, as we find that Ugandan and non-Ugandan experts think about democracy in different ways. However, nationality itself also seems to matter. In a second survey, we assess levels of trust in the scores provided by these two types of raters as well as the official Freedom House rating. We find that international experts have significantly lower trust in ratings provided by Ugandan experts than they do in international expert ratings or the official Freedom House rating. Meanwhile, Ugandan experts report the highest levels of trust in the Ugandan experts’ rating, and the least trust in the official Freedom House rating, a difference of which was statistically significant. These findings illuminate how expert assessments may vary depending on the identity of the expert, which may in turn have implications for the quality of these ratings, as well as how they are perceived and used by international and domestic publics.

Authors

• Sarah S. Bush, Yale University

• Melina Raquel Platas Izama, NYU Abu Dhabi