Consequences and Side Effects of Testing: Their Role in Validity and the Meaning of Test Scores

• In Event: Exploring Unintended or Inappropriate Interpretations, Uses, and Consequences of Test Scores

Abstract

Ultimately, measures in research, testing, assessment, and evaluation are used, or have implications, for ranking, intervention, feedback, decision-making, or policy purposes. Explicit recognition of this fact brings the often-ignored and sometimes maligned concept of consequences to the fore (Zumbo, 2009). Given that measures have personal and social consequences and impact, it becomes important to evaluate the intended consequences and unintended side effects of measurement when validating the inferences and uses made from tests. The purpose of this presentation is to apply Hubley and Zumbo’s (in press) recently developed integrated framework of validity and validation to the matter of consequences of educational test use.
The Hubley-Zumbo integrated framework for test validation expands on the concept of consequences while also situating them in their proper place relative to other types of validity evidence. Intended consequences and unintended side effects become but one of many different forms of evidence that can be presented when evaluating the interpretation and use of test scores.
Although some writers have argued that social consequences have no place in validity, their argument tends to be based on a misconception that social consequences are about test use and, in particular, test misuse or inappropriate use (Mehrens, 1997; Popham, 1997). Like Messick (1998), we do not view test misuse as an aspect of (social) consequences in the context of validation. While test misuse is an important concern, it is irrelevant to discussions of score meaning and thus outside of construct validity and the validation process. Instead, it is the consequences of legitimate test use that may influence test score meaning and inference and what makes consequences relevant to the validation process. By presenting this new integrated framework and discussing several educational testing cases and contexts, we will illustrate why it is important that consequences be considered a part of validity and validation. We argue that it is helpful to use different terms to distinguish between consequences that are intended and those (side effects) that are not. Furthermore, we argue that it is useful to consider personal consequences and side effects and not just social ones alone.
The role of consequences in validity and validation currently is a topic of great controversy and discussion in the measurement field. Much of the discussion against the inclusion of consequences in validation focuses on (a) test misuse, and (b) concerns about the feasibility and burden of incorporating consequences as an expected element of the validation process. In our new framework, we correct some misconceptions about consequences, expand our understandings of this concept, and situate consequences and side effects in such a way that incorporating them becomes more feasible and less daunting.

Authors

• Bruno D. Zumbo, The University of British Columbia

• Anita M. Hubley, The University of British Columbia