Examining the Validity Evidence for SAT/ACT in Admissions: Framework and Methods for Validation Review

• In Event: Do SATs/ACTs Inform Equitable Admissions and Scholarship Decisions for the 21st Century? A Validation Review

Abstract

Purpose

In this paper researchers introduce the context of the validation review, positioning it within the context of Smith v. Regents (2019), the four pandemics (i.e., COVID-19, racism, the threat of economic collapse, and impending environmental catastrophe; see Ladson-Billings, 2021), and the literature on validity and validation (AERA et al., 2014; Kane, 2013).

Perspectives

Per the Standards (AERA et al., 2014), validity is not a property of tests but the interpretations and uses of tests; hence, validation begins with a definition of an intended interpretation/use argument (IUA; Kane, 2013) being validated. Researchers systematically reviewed articles published in high-quality journals over the past 50 years to interrogate validity evidence that supported (or challenged) the following IUA: SAT and ACT tests accurately, consistently, and fairly measure college readiness and can predict college success; thus, SAT and ACT scores support valid college admissions and merit-based scholarship decisions. Researchers used a framework for validity evidence developed in which evidence of reliable, precise, and fair methods and measurement is fundamental, as are five additional sources of evidence also described in the Standards (see Figure 1; AERA et al., 2014; Author(s), 2016).

Methods

Researchers consulted the 2018 Journal Citation Reports® (JCR) to select journals of sufficient quality. Any journal rated in the top JCR quartile qualified for inclusion in this study. Researchers used EBSCOhost to search for articles published in these journals containing the SAT or ACT along with mentions of admissions or scholarships published from 1969 to 2019. Figure 2 displays the number of records found and excluded at each stage of the review process per PRISMA guidelines (Moher et al., 2009). Researchers developed a review instrument by which two reviewers rated the evidence provided by extent to which author(s) of each paper were supportive or challenging the SAT or ACT on a five-point scale from 5 = supportive to 1 = challenging (see blinded instrument here: http://bit.ly/38Xse8O). Researchers then collectively analyzed and synthesized their quantitative ratings and qualitative notes.

Results

The overall weight and direction of validity evidence provided was positioned as neither supportive nor challenging to the valid use of SAT and ACT scores as per the aforementioned IUA, though ratings varied by area of the framework and positions taken by author(s) of each paper reviewed. Table 1 displays mean support ratings, capturing how authors positioned the evidence provided. Each subarea of evidence is discussed in subsequent symposium papers, along with qualitative syntheses and discussions of the nature of the evidence presented therein.

Significance

Beyond the “Scholarly Significance” of this panel described prior, the significance of this particular paper is primarily methodological, as per the validity framework researchers developed and used to conduct this systematic review, as also based on the eminent Standards (AERA et al., 2014) and Kane’s (2013) contemporary view of validity. This framework, also adopted and refined from a prior study (Author(s), 2016) is, in other words, significant in and of itself, especially for those who are generally interested in examining evidence of validity.

Authors

• Audrey Amrein-Beardsley, Arizona State University

• Matthew Ryan Lavery, South Carolina Education Oversight Committee