Developing an Instrument With Validity in Mind: A More Accessible Model for Validation by Design

• In Event: Informing Perspectives on Validation: Narratives From Mathematics Education Scholarship

Abstract

The Standards (AERA, APA, & NCME, 2014) open with a statement that “educational and psychological testing and assessment are among the most important contributions of cognitive and behavioral sciences to our society” (p. 1), and goes on to devote its first chapter to validity and validation, identifying it as “the most fundamental consideration in developing tests and evaluating tests” (p. 11). Despite the perceived importance, however, validity and validation remain underused in practice. In a review of 121 instruments used in projects funded by the National Science Foundation’s DRK-12 program, Minner and colleagues (Minner, Erickson, Wu, & Martinez, 2012; Minner, Martinez, & Freeman, 2012) found that only 67 of those instruments (55%) had any validity evidence available for them. At present, the predominant model of validation is a process of argumentation in which the proposed interpretations and uses of a test are supported with sufficient, appropriate validity evidence (e.g., Kane, 2006, 2013, 2016; Schilling & Hill, 2007), but a recent review of 83 papers advancing argument-based validation methods found little clear consensus on how to construct such arguments and conduct the important work of validation (Symposium author_D, 2019). By contrast, the Standards (AERA et al., 2014), which represent 60 years of consensus in the field of educational and psychological testing, provide detailed standards regarding the evidence used in validation, but little clear guidance on how to construct the validity argument or to conduct validation. Further, although the Standards (AERA, et al., 2014) state that “validation logically begins with an explicit statement of the proposed interpretations of test scores” (p. 11), it does not specify when validation should occur relative to test development.

The present paper seeks to address the underutilization of argument-based validity and validation in practice by presenting a more broadly accessible approach; positioning validation as a design consideration addressed during the development of an educational or psychological instrument. Building on the recommendations of participants at an NSF-sponsored conference of experts in mathematics education, psychometrics, and validation, we describe the development of a detailed interpretation and use statement (IUS) as a first step in the planning and preparation of an instrument. The IUS is further expanded through a series of development questions designed to focus developers’ attention on specific aspects of the proposed interpretation and use of the test that must be supported with validity evidence. The development question approach may offer some resolution to the ongoing tension between validity scholars who argue for the flexibility offered by unstructured approaches to validation built on informal logic (e.g., Kane, 2006, 2013, 2016) and those scholars who argue for more structured approaches to validation built on formal logic (e.g., Mislevy, 2004, 2016; Schilling, 2007; Schilling & Hill, 2007); being both accessible to discipline based researchers who may not have extensive psychometric training, and flexible enough to support multiple measurement purposes in a variety of contexts. Examples in mathematics education contexts will be shared.

Author

• Matthew Ryan Lavery, South Carolina Education Oversight Committee