Judgment Validity: It’s Time to Move Beyond the Usual Debate

• In Event: Topics in Test Validity: Theory and Practice

Abstract

The author contends that current debate about statistical versus impressionistic combination of applicant information in admissions is a false dichotomy between the statistical benefit of achieving greater predictive accuracy (greater criterion-related validity) versus the impressionistic benefit of achieving greater student diversity (greater consequential validity). Researchers instead should explore the possibility of using judgment approaches that simultaneously achieve both greater accuracy and greater diversity. Two main ways in which impressionistic judges could outperform statistical models – by using non-linear functional forms or “broken legs” – face theoretical obstacles, but discussion about whether and how these obstacles might be overcome has been inadequate. Furthermore, fungible weights research suggests that impressionistic judges theoretically could identify weights promoting diversity with little sacrifice in predictive accuracy.

Author

• David Marc Klieger, Educational Testing Service