Examining Response Process Validity Evidence on SAT/ACT Use in College or University Admissions

• In Event: Do SATs/ACTs Inform Equitable Admissions and Scholarship Decisions for the 21st Century? A Validation Review

Abstract

Purpose

For this paper researchers introduce the notion of response process sources of validity evidence (AERA et al., 2014; Messick, 1989, 1995), again, within the same context of the same review, but with 18 of the 74 papers (24% of the sample of papers reviewed) containing claims about the SATs and ACTs and IUA of interest.

Perspectives

According to the Standards (AERA et al., 2014), response process sources of validity evidence are meant to capture how well the construct or performance of any target domain matches the responses or performances measured to inform the IUA at issue. This area of validity is primarily concerned with issues of irrelevant method variance that may limit the validity of the proposed interpretations of the test (Messick 1989, 1995).

Methods

Again, this study was part of a larger project described prior. In total, 18 papers (24%) were coded using the aforementioned review instrument as each of the 18 papers contained validity evidence related to response processes.

Results

Authors of these 18 studies provided direct or cited evidence that was positioned as neither supportive nor challenging of the IUA. The most common theme among articles focused on different types of test preparation (n = 9; 50%), including increasing test-taking skills (Alderman & Powers, 1980; Messick & Jungeblut, 1981) and increasing test and item-type exposure through practice sessions and repeated retaking of tests (Becker, 1990a; DerSimonian & Laird, 1983; Isaacs, 2001; Slack & Porter, 1980a; Walpole et al., 2005). Related, Alon (2010) and C. K. Jackson (2010) explored issues of ethnicity involving coaching or training interventions.

In contrast, the four studies in which researchers focused on test preparation but that were supportive of the IUA promoted simple drill and practice (R. Jackson, 1980), the blending of content review with the use and discussion of sample items (Byrnes & Takahira, 1994), and the issue of maturation regardless of the coaching or training engaged (Brody & Benbow, 1990). In addition, authors of two studies highlighted methodological issues as potential reasons why findings were inconsistent among validity studies pertaining to coaching or training (Alderman & Powers, 1980; DerSimonian & Liard, 1983). Finally, authors of five studies (Aguinis et al., 2016; Becker, 1990b; Bridgeman & Wendler, 1991; Fagan & Holland, 2009; Freedle, 2004) explored more traditional aspects of response processes including cognitive demand, response patterns, task completion, and the use of cognitive interviews.

Significance

Response process evidence of validity is an important issue to explore regarding the intended use of SAT and ACT test scores. This is especially true when examining potential differences in access to coaching or training, as well as the relationship that authors of some of the studies reviewed found between certain types of test preparation and subsequent growth in students’ SAT and ACT scores.

Author

• Deborah M. La Torre, University of California - Los Angeles