Examining the Validity Evidence for SAT/ACT in Admissions Reliability/Precision and Internal Structure

• In Event: Do SATs/ACTs Inform Equitable Admissions and Scholarship Decisions for the 21st Century? A Validation Review

Abstract

Purpose

For this paper, researchers analyzed 13 papers (18% of the sample of 74 papers reviewed) in which authors directly studied or mentioned (1) the reliability/precision of the SATs or ACTs, and (2) validity evidence based on the tests’ internal structures.

Perspectives

As per the Standards (AERA et al., 2014), the term reliability/precision denotes “the consistency of the scores across instances of the testing procedure” (p. 33). Internal structure validity concerns “the degree to which the relationships among test items and test components conform to the construct on which the proposed test score interpretations are based” (AERA et al., 2014, p. 16). For example, if a test is intended to include several components that are each homogeneous but also distinct from each other, statistical analyses of the interrelationships of test items can be used to assess the extent to which such a posited structure is maintained. Some types of reliability and differential item functioning (DIF) also fall within the scope of internal structure. Researchers, however, exclude the discussion of DIF, which is discussed in Paper #6 as an integral part of test fairness.

Methods

As part of the larger project described prior, for this part of the study researchers applied the methods described in Paper #1, examining evidence of reliability/precision and both tests’ internal structure, again, using the aforementioned review instrument and the guidelines written into the Standards (AERA et al., 2014) that directly related to both sources of evidence.

Results

Reliability/Precision. Authors of five reviewed papers (7% of the sample) provided evidence related to the reliability/precision of the SATs and ACTs, which they positioned as somewhat supportive of the aforementioned IUA. The reported reliability values of the operational SATs or ACTs or their subtests ranged between .89 and .93 (Coyle, 2006; Stricker et al., 1993). In short, this review suggests there is solid evidence that the reliability of SAT and ACT scores typically exceeds .90 which, as per Aguinis et al. (2016), exceeds “recommended reliability standard for the majority of purposes cited in organizational research” (p. 5) and should be held as very high in any educational and psychological measurement milieu.

Validity Evidence Based on Internal Structure. Authors of eight reviewed papers (11% of the sample) provided evidence related to the internal structure of the SATs and ACTs, which they positioned as somewhat supportive of the use of these tests in college admissions decisions. With further in-depth qualitative review, researchers concluded that the handful of studies in which such evidence was available seemed benign and did not challenge the validity of the IUA.

Significance

Reliability is a necessary requirement for the validity of a test. A test must be reliable before being considered valid for any IUA. Internal structure is one of the five sources of validity evidence entailed by the Standards (AERA et al., 2014). Hence, the review of these two aspects is essential to the review of validity of the SATs and ACTs for the IUA.

Author

• Yi Zheng, Arizona State University