Building a Conceptual Bridge Between Theories of Action and Interpretative Arguments

• In Event: Roundtable Session 117
  In Roundtable Session: Searching for Clarity With Theories of Action and Validity Arguments: The Place of Consequence

Abstract

The argument-based approach to validation has a long history culminating in Kane’s (2006) chapter in the fourth edition of Educational Measurement. Unfortunately, the conceptually challenging work of creating and validating interpretative arguments has not moved from the theoretical literature to practical operations. Further, in spite of the Kane’s (2006) example of NCLB tests in his chapter, the interpretative argument appears focused on the evaluation of the defensibility of inferences from fairly straightforward testing situations. The challenges of validating assessments used in complex policy contexts have led researchers and evaluators to search for more practical approaches for initiating validity evaluations.

Several recent projects have worked with state leaders to design and carry out validity evaluations (Author & Author, 2006; Author & Author, 2009; Author & Author, 2011). These projects used Kane’s argument based approach as a theoretical foundation and supported states in developing interpretative arguments to structure the evaluations, but this proved to be a conceptually challenging enterprise. Therefore, these projects relied on theories of action as a starting point for developing a more formal interpretative argument. Theories of action were used to lay out the network of assumptions, claims, mechanisms, intended outcomes, and potential consequence of the testing programs. These theories of action are broader than an interpretative argument, particularly one that uses Kane’s (2006) model of scoring, generalization, extrapolation, and decision, but they also serve as a useful starting point for helping states begin to think about validity evaluations.

While the use of these theories of action have demonstrated a high degree of utility in advancing validity evaluations of state testing programs, it is time to step back to examine and strengthen the conceptual underpinnings in the interaction of theories of action and interpretative arguments. Recent work in this area (Author, 2010; Author et al., 2011) has pushed our thinking about the nature of the relationship between these two important heuristic devices. This paper take the position that interpretative arguments are located within a broader theory of action and uses this perspective to examine the role and place of both intended and unintended consequences of the testing program. The paper further argues that consequential evidence is central to validation, but explores the struggles that both theorists and practitioners have had with how best to incorporate such evidence into validity evaluations. The paper claims that a theory of action is critical for evaluating assessments and assessment systems that serve key policy functions, but that consequential evidence serves as important link between the interpretative argument and the theory of action.

Authors

• Scott F. Marion, National Center for the Improvement of Educational Assessment, Inc.

• Marianne Perie, Center for Educational Testing and Evaluation

• Paul D. Nichols, Pearson