Validating Inferences Under High-Stakes Conditions: Addressing Unintended but Expected Consequences of Accountability

• In Event: Exploring Unintended or Inappropriate Interpretations, Uses, and Consequences of Test Scores

Abstract

Recent years have seen a fundamental change in the uses of large-scale testing. Diagnosis has been supplanted by accountability, and the pressure to raise scores has risen from one decade to the next. Research has revealed two unintended effects: inappropriate test preparation and score inflation (e.g., Jacob, 2005; Koretz & Barron, 1998; National Research Council, 2011; Stecher, 2002). In some cases, the result has been a very large bias in scores that undermines evaluation of both overall gains and the relative performance of teachers, schools, and districts. These findings pose critically important validity questions, particularly given that the primary purpose of accountability testing is to improve instruction and that inferences about gains (or performance relative to targets) are currently among the most important. Nonetheless, evaluation of these aspects of validity remain uncommon. The dearth of validation studies may reflect several factors. One is political: under the current system, it is usually in no one’s interest to evaluate whether scores have become inflated. A second is a shortage of useful available data, such as scores from a second, less inflated audit test. A third is the burden and costs of collecting needed data. However, professional tradition may also play a role: neither the threats to validity posed by high-stakes uses nor the evidence needed to evaluate it are commonly addressed, for example, in texts or the technical reports of testing programs.
This paper addresses issues that arise in evaluating validity under high-stakes conditions. It begins with illustrative empirical examples of effects of high stakes on both instruction and the validity of scores. It provides a framework for evaluating VIHS (Koretz, McCaffrey, & Hamilton, 2001) and explains why traditional validity evidence is insufficient under high-stakes conditions. It then describes types of evidence that may be used and sources for that evidence, for example, extant audit tests and self-monitoring assessments (Koretz & Beguin, 2010). A final section discusses options for making evaluation of validity under high-stakes conditions more routine and which parties in the testing enterprise might share responsibility for it.

Author

• Daniel M. Koretz, Harvard University