The Role of Interrater Reliability in Systematic Reviews

• In Event: Doing Systematic Reviews in Crime Prevention

Abstract

Interrater reliability (IRR) is an integral part of a systematic review process. An essential component of systematic searches is transparency and replicability of the search process. Good empirical studies with two or more coders rely on IRR to establish the reliability of the coding by demonstrating a high degree of agreement between coders. IRR tends to be reported using approved statistical tests, however the process by which an acceptable degree of agreement is reached is rarely explored in detail. During the course of generating a list of systematic reviews on What Works in Crime Prevention, we found that the process was more complicated and subjective than previously reported. This paper describes the difficult process of achieving consensus among different coders throughout the evolution of a codebook. It highlights the human element and psychological processes that affect coder behaviour and decision making which in turn has an impact on the IRR. It discusses the results of several IRR tests conducted during the course of the project and describes how coding improved and evolved as a result.

Authors

• Tanya Le Sage, University College London

• Jyoti Belur, University College London

• Lisa Tompson, University College London