Instantiating the Validity Argument Framework: Evaluating the Validity of the Uses of Universal Screeners

• In Event: Argument-Based Approaches to Measure Validation Situated in Mathematics Education

Abstract

Objectives: Despite the call for an argument-based approach to validity over 25 years ago, few examples exist in the published literature. The purpose of this manuscript is to illustrate the argument-based approach to validity for the uses and interpretations of a universal screener for middle-school mathematics. Specifically, a universal screening assessment system was created for use within a RtI framework to help middle school teachers support students’ algebra readiness. This universal screener is a computer-based multiple-choice assessment that takes approximately 20 minutes to complete, and is administered three times per year (fall, winter, and early spring).

Theoretical Framework: Universal screeners are formative assessments that form an integral part of a comprehensive assessment framework for implementing Response to Intervention (RtI). Results from universal screeners help teachers make instructional decisions early in the learning process to prevent and remediate skill gaps. Data are intended to help teachers (1) identify students who are at-risk for future failure in the domain and then (2) determine the level of intensity of supplemental instructional support that may help at-risk students reach their learning goals. Because of the important use of universal screener results, sufficient validity evidence should substantiate the trustworthiness and meaningfulness of the results from making these decisions.

A validity argument is the process of creating an evidence-based case for the intended interpretations and uses of the observed score. Messick (1995) summarized the importance of this process, stating, “score validation is empirical evaluation of the meaning and consequence of measurement. As such, validation combines scientific inquiry with rational argument to justify (or nullify) score interpretations and use” (p. 5). The extent to which a score-based decision is justifiable depends on the clarity and coherence of the accumulated evidence (Kane, 1992, 2006, 2013).

Modes of Inquiry: First, this paper presents an interpretive and use argument (IUA) that outlines the inferences and assumptions leading from the score on the universal screener to the intended uses and interpretations. The inferences are sequenced following Kane’s (1992, 2006) structure to progress from scoring to generalization to extrapolation inferences. The assumptions underlying each inference serve as the testable hypotheses from which evidence is collected in a validity argument.

Data Sources: Second, this paper presents a validity argument by gathering evidence designed to address the hypotheses outlined in the IUA. Evidence is derived from procedural sources collected during the test development process, psychometric analyses from a large-scale field test, descriptive and correlational studies conducted during pilot testing, and interviews and observations of teachers using the results to make classroom-based decisions.

Substantiated Conclusions and Implications: The defensibility of the evidence is evaluated and an overall evaluation of the validity of the interpretations and uses of the universal screener is provided. Implications for practitioners using the universal screener are proposed, and the findings are generalized to inform test developers undergoing similar efforts.

Author

• Leanne R. Ketterlin-Geller, Southern Methodist University