Theory of Action as a Framework for Evaluating an Assessment Program

• In Event: Roundtable Session 117
  In Roundtable Session: Searching for Clarity With Theories of Action and Validity Arguments: The Place of Consequence

Abstract

Author et al. (2011, p. 6) propose a reformulation of Kane’s (2006) notion of interpretative argument that includes theory of action:

"The interpretive argument specifies the proposed interpretations and uses of scores by laying out a chain or network of inferences and assumptions leading from the observed performances to the conclusions and decisions based on the scores. To the extent that the assessment itself is intended to change individuals or institutions, the interpretive argument specifies a theory of action for the assessment, including the impact claims and the mechanisms though which that impact is expected to occur."

Including the theory of action within the interpretative argument combines within the same framework elements that would have traditionally been considered the province of educational evaluation and those that would have more typically been seen as validation. The combined set of elements includes:

• The intended effects of the assessment system;
• The components of the assessment system and a logical and coherent rationale for each component, including backing for that rationale in research and theory;
• The interpretive claims that will be made from assessment results;
• The action mechanisms designed to cause the intended effects; and
• Potential unintended negative effects and what will be done to mitigate them.

The research plan to validate the assessment program would, therefore, give attention to such questions as:

• Is the theory of action logical, coherent, and scientifically defensible?
• Was the assessment system implemented as designed?
• Were the interpretive claims empirically supported?
• Were the intended effects on individuals and institutions achieved, and did the postulated mechanisms appear to cause those effects?
• What important unintended effects appear to have occurred?

Each such question presumes a set of more specific sub-questions derived from the details of the interpretive argument and its subsumed theory of action for the assessment program of interest.

This paper will use as an example, the theory of action underlying an innovative assessment-program model intended to advance thinking and practice in K-12 summative and formative assessment (Author, 2010). According to Author (2010, p. 147), this model, and the research done with the assessment prototypes generated from it, might offer useful input for the Race to the Top Assessment Program consortia. This paper will focus on using the theory of action to identify, and to suggest how to address, specific questions related to the interpretive claims and intended effects of this assessment-program model.

Author

• Randy E. Bennett, ETS