Better Human Annotation: Methods to Decrease Random and Systematic Errors

• In Event: Preprocessing, Annotation, and Interpretation in Computational Social Science

Abstract

While much of social science research relies on human-annotated data such as conflict events and regime types, there is little research on evaluating and improving the quality of human-annotated data. In general, scholars have been using multiple coders for quality control and a majority rule to solve disagreements. The underlying assumption is that coders make random errors. Therefore, by averaging multiple coders' answers, we will get unbiased or accurate coding. Unfortunately, human coders can also make systematic errors, especially when the tasks are subjective. Instead of assuming all errors are random, we argue that it is possible to model or measure both types of errors using the characteristics of human coders. For example, a human coder's levels of literacy and attention are likely to influence the size of the random errors he/she makes. On the other hand, depending on the particular tasks, a person's ideology and group biases might systematically distort the coding towards a particular direction. In this project, we evaluate how coders' characteristics influence their accuracy and propose a refined coding scheme and a modeling approach to mitigate the bias.

Authors

• Yaoyao Dai, University of North Carolina, Charlotte

• Robert Pinka, Council For Children's Rights

• Benjamin J Radford, University of North Carolina at Charlotte