Using Validity Arguments to Evaluate the Technical Quality of Local Assessment Systems

• In Event: Roundtable Session 4
  In Roundtable Session: Validating Local Assessments

Abstract

Objectives
The purpose of this presentation is to explore the demonstration of technical quality in a local assessment system by using a validity argument framework. Previous efforts have introduced validity argumentation to alternative assessment systems for students with disabilities. This presentation goes a step further by addressing the use of validity arguments in local assessment systems that are intended to supplement or supplant state systems.

Theoretical framework
Messick (1989) and the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999) conceptualized validity as pertaining to inferences, drawn from test scores, about a construct, and discussed gathering evidence to evaluate intended interpretations. Recent validity scholars (e.g., Kane, 2006) have extended the notion of gathering validity evidence by framing validity in terms of a chain of interpretive arguments. Each interpretive argument moves from an observation to a claim. One interpretive argument’s claim becomes the observation in a subsequent interpretive argument. Collectively, this chain provides contextualized explanations of test scores. Applied to the evaluation of a local assessment system, interpretive arguments allow for judgment of the extent to which the local system maintains appropriate levels of technical quality.

Methods and results
Artifacts from previous ventures into local assessment systems by other states, the meetings of a state local assessment validation advisory committee, and cases of developing validity arguments for other assessments (e.g., TOEFL, Chapelle, Enright, & Jamieson, 2010) served as primary sources of reference. Establishing criteria for technical quality in a local assessment system required constant consideration of feasibility while adhering to the mandates of a sound validity argument for judging student proficiencies. Given limited resources common to local school districts, framing a validity argument resource for local assessment systems focused on front-end design considerations, placed in a comprehensible context that demonstrates how sound design leads to defensible conclusions. To move from abstract pontificating about score inferences to concrete guidance in assembling a validity argument, the author advances a framework for specifying interpretive arguments that reveal aspects of technical quality. Claims reflecting necessary precursors to valid score interpretations (e.g., The local assessment system maintains an adequate level of rigor) provide the structure. Local school districts, with the assistance of exemplars of supporting evidence, are tasked with providing the evidence and backing for these claims. This framework allows system evaluators to render judgments based on the clarity, coherence, and plausibility of the interpretive arguments.

Significance
The validity argument framework has not been attempted widely in statewide local assessment systems. A benefit to the validity argument framework is that it allows for evaluations to be made in the absence of a strong, unified theory of an underlying construct. Such a situation describes the context of much educational testing. In the case of local school districts implementing an assessment system, the validity argument framework provides clear guidance, and leads local professionals toward thinking more deeply about the purposes of the assessment system and the assumptions upon which judgments about students depend.

Authors

• Chad M. Gotch, Washington State University

• Marianne Perie, Center for Educational Testing and Evaluation