Testing the Reliability, Validity and Equity of Terrorism Risk Assessment Tools

• In Event: Strategically Countering Terrorism

Abstract

Current efforts to develop scientifically rigorous and operationally relevant research related to the risk assessment and threat management of the terrorist remains worryingly underdeveloped. An increasing number of risk assessment tools are being developed in the published literature. These include the Terrorist Risk Assessment Protocol 18, the Identifying Vulnerable People tool, the Extremism Risk Guidelines, and the Violent Extremist Risk Assessment guide. We then put these tools to the test against a library of cases that include true positives (actual violent extremists) and true negatives. The intention here is to evaluate commonly used risk assessment tools by comparing their predictive validity, reliability, equity and cost. We will use multiple measures to assess inter-rater reliability (percentage agreement, Kappa, ICC). We will then check which risk assessment protocols consistently score highly across each inter-rater reliability measure. Because we have built in false positives and counter-factual cases, we will have the ability to test for validity (Dispersion Index for Risk, Receiver Operator Curves, diagnostic odds ratios; and the Area under the Curve). Because the cases will range across ideologies and terrorist type, we will be able to test equity (e.g. how well the tool applies to different categories of violent extremists).

Authors

• Paul Gill, University College London

• Verity Buckley, University College London