Review of the Evidence for SAT and ACT Test Content Validity

• In Event: Do SATs/ACTs Inform Equitable Admissions and Scholarship Decisions for the 21st Century? A Validation Review

Abstract

Purpose

In this paper, researchers introduce the notion of test content-related evidence of validity (AERA et al., 2014; Kane, 2013) within the context of a review of 17 of the aforementioned 74 papers (23% of the sample) containing related claims about the SATs and ACTs.

Perspectives

As per the Standards (AERA et al., 2014), content-related evidence of validity refers to the extent to which the content of the SATs or ACTs support the aforementioned IUA (see Paper #1; see also Kane, 2013) and is primarily concerned with the relationship between the content of the tests and the constructs tests are purported to measure (Cronbach, 1971; Cureton, 1951; Kane, 2013).

Methods

Again, of the 74 articles selected for the final review, 17 papers (23% of the sample) contained test-content validity evidence which researchers in charge of this part of the study coded according to the aforementioned review instrument and the guidelines written into the Standards (AERA et al., 2014) that directly related to content-related evidence of validity.

Results

Researchers found that the authors of these 17 studies provided direct or cited evidence which somewhat challenged the use of SATs and ACTs in college admissions decisions. Specifically, researchers found that a majority of study authors challenged the content validity component of the IUA based on (1) both tests’ coachability (Slack & Porter, 1980a, 1980b); (2) the incompleteness of both tests in measuring college preparedness, which results in underestimation of the construct for minorities and women (Becker, 1990b; Bridgeman & Wendler, 1991; Britton & Tesser, 1991; Fagan & Holland, 2009; Freedle, 2003; Keiser et al., 2016; Sternberg, 2006; Walpole et al., 2005); and (3) the unfeasibility of assuming that a single test can address the diverse priorities of states and universities (Atkinson & Geiser, 2009). A minority of study authors acknowledged these concerns but considered them overstated (Jackson, 1980; Messick & Jungeblut, 1981), arguing in support of both tests’ continued validity for college admissions (Breland & Griswold, 1982; Linn, 2009).

Significance

Content-related evidence of validity, again, is an important consideration in educational and psychological measurement as per the Standards (AERA et al., 2014). For this part of the larger study, researchers identified and examined particular studies in which authors raised substantive objections or praise regarding the content-related evidence of validity surrounding the use of the SATs or ACTs as measures of readiness for a broad array of higher education institutions with a diverse range of educational aims. In light of the findings presented within this section of the study, the movement to limit the assumed authority of the SATs and ACTs as measures of high school learning and college readiness seems to be both appropriate and evidence-based.

Authors

• Norman P. Gibbs, Arizona State University

• Margarita Pivovarova, Arizona State University