Affordances and Constraints of Two Approaches to Validation Arguments

• In Event: Argument-Based Approaches to Measure Validation Situated in Mathematics Education

Abstract

In spring of 2017 measurement and mathematics education researchers came together to examine argument-based approaches to measure validation within the context of mathematics education. Several frameworks were presented as potential approaches with a focus on articulating the claims and assumptions within validation arguments. The purpose of this paper is to compare and contrast two of these frameworks by articulating two different validation arguments for the same instrument. Our goal is to identify the affordances and constraints provided these two frameworks.

The two frameworks are:
The interpretative argument aspect of Kane’s (2004) observed performance to interpretation for use approach the first step of which is stating the chain of assumptions and inferences that start at the process of converting an observed performance to a score for an item and ends at the interpretation of the test score for a particular use.
The sources of validity from the Standards for Educational and Psychological Testing (2014) can also be used to identify and articulate the claims and assumptions inherent in the stated score interpretation for proposed uses.
Each framework will be used to independently articulate claims and assumption for the instrument described below.


The instrument is the Diagnostic Assessment of Proportional Reasoning (DAPR) which measures student composed unit and multiplicative comparison conceptions in proportional reasoning situations. The DAPR is a 20 item fill-in-the-blank assessment available in three equated forms and administered by classroom teachers. The assessment content is targeted at middle grades standards in the Common Core and several studies generating aspects of validity evidence using students in grades 6-9 have been conducted. A student’s DAPR score can be interpreted in relation to a learning trajectory of composed unit and multiplicative comparison understanding and used by classroom teachers to identify instructional scaffolds for students.

The claims and assumptions across the two frameworks will be generated independently and then the resulting arguments compared. The goals is to identify the affordances and constraints of argument-based frameworks. For example, initial results from discussions at the Validity Evidence for Measurement in Mathematics Education conference indicates an affordance of Kane’s framework is the press for a coherent chain of reasoning but because the argument development process differs from the typical instrument development process, it is often more difficult for developers to articulate than more traditional approaches, such as the sources of validity.

Currently, validation work in mathematics education tends to focus on presenting isolated aspects of validity evidence. However, this evidence is seldom associated with a specific claim or assumption and is rarely situated within a comprehensive validation argument. This paper can serve as an example within the mathematics education for structuring an argument that identifies specific claims and assumptions inherent in a particular score interpretation and foster discussion around the two frameworks and more generally about an argument-based approach to validation.

Authors

• Michele Carney, Boise State University

• Carl Siebert, Boise State University

• Angela Rae Crawford, Boise State University

• Keith W. Thiede, Boise State University

• Richard D. Osguthorpe, Boise State University