Evaluation of the Interrater Reliability (IRR), Conditional IRR, and Validity of a Juvenile Risk Assessment

• In Event: Evaluating CJ Interventions: Facial Recognition, Risk Assessment, and Implementation Science

Abstract

Criminal justice agencies sometimes focus attention only on the predictive validity of risk assessments. While understandable given its importance, examining only predictive validity neglects the fact that interrater reliability (IRR) is a necessary precursor. We intend to present results from an evaluation of IRR, conditional IRR, and gold standard validity from a juvenile risk assessment tool. Though IRR is sometimes given attention by agencies, interrater validity and conditional IRR are less commonly evaluated, and perhaps not well-known. Interrater validity considers the agreement among raters when a subject's "true" score is known, as determined by a gold-standard rater. This form of validity is essential to ensure that the raters' evaluations are not only consistent with one another but also accurate in terms of what is considered “true”. Conditional IRR analysis takes this a step farther and explores why IRR or interrater validity issues might have occurred by examining problematic response options within items. This approach helps identify categories where consensus is challenging for raters, often as a function of training issues or response options that are not well-differentiated. Results are discussed in terms of their potential impact on predictive validity.

Authors

• Kort Prince, University of Utah

• Christian Sarver, University of Utah