Paper Summary
Share...

Direct link:

A Framework and Instrument for Systematic Reviews of Validity Evidence

Sat, April 9, 10:35am to 12:05pm, Marriott Marquis, Floor: Level Two, Marquis Salon 1

Abstract

Purpose

Validity is a complex, multifaceted construct requiring several types of evidence from multiple sources to establish it (American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME), 2014; Kane, 2013). As such, researchers may not find sufficient evidence to determine the validity of a particular interpretation or use of test scores in a single study. A validation review, defined as a systematic review of validity evidence in the peer-reviewed literature of the field, may be better suited to the task. Researchers in this study present a framework and a related data collection instrument to support such a validity review.

Perspectives

Test and measurement output are neither valid nor invalid, rather the interpretations and uses derived from such output are that which must be validated (AERA et al., 2014; Kane, 2013; Messick, 1995). Thus, the first and most important step in validation is clear articulation of an interpretation/use argument (IUA; Kane, 2013; see also Brennan, 2013; Haertel, 2013), and a well-formulated IUA may be validated through the integration of “various strands of evidence into a coherent account of the degree to which existing evidence and theory support the intended interpretations of test scores for specific uses” (AERA et al., 2014, p. 21).

Methods

Researchers consulted the extant literature on validity and validation (e.g., AERA et al., 2014; Cronbach & Meehl, 1955; Kane, 2013; Messick, 1989) to develop a framework and instrument to support their validation review. Researchers collaboratively discussed, developed, and revised the framework, and they developed, piloted, and revised the accompanying instrument before using it in their validation review. A full version of the final instrument (blinded for peer-review) is available at https://goo.gl/ECm1mT.

Results

The framework illustrated in Figure 1 supports the synthesis and evaluation of the compiled validity evidence and current theory related to a specific IUA. The IUA being validated rests directly upon the sound measurement and methodology used to generate the scores being interpreted for intended uses (AERA et al., 2014; Kane, 2013), both of which rest in turn on validity evidence from five sources: 1) test content, 2) response processes, 3) internal structure, 4) relations to other variables, and 5) related consequences (six areas of validity evidence including the aforementioned measurement and methodology; AERA et al., 2014; Kane, 2013). While it might not be necessary to collect exhaustive evidence in all areas, each of the elements in the framework represents a legitimate basis upon which to support or challenge the validity argument for whatever IUA might be under investigation.

Significance

Figure 1, and the validation instrument researchers developed to accompany it (see link above), provide a framework with which to organize and assess compiled validity evidence related to a specific IUA. The framework and instrument are significant to the field as tools to support future validation efforts. This will be discussed next, as used/applied in researchers’ review of 92 articles covering 100 years of literature surrounding the use of test scores to evaluate teachers (see Papers 2-4 forthcoming).

Author