Examining the Arguments Surrounding the Argument-Based Approach to Validation: A Systematic Review of Validation Methodology

• In Event: Argument-Based Approaches to Measure Validation Situated in Mathematics Education

Abstract

As early as Descartes (1637/1970), logic and reason have been positioned as tools for individuals to advance their own understanding. By contrast, argumentation is an interactive, social exercise used for persuasion, collective cognition, and to advance shared knowledge (Mercier & Sperber, 2011, 2017). When one advances an argument, subjects it to the tests and challenges of others, and responds to questions and counterarguments, one’s thinking improves (Mercier & Sperber, 2017). Through argumentation, groups produce correct solutions more often than individuals (Moshman & Geil, 1998) and individual accuracy improves as well (Castelain, Girotto, Jamet, & Mercier, 2016). Since it was formally introduced by Kane (1990, 1992), the argument-based approach to validation has been promoted in the field of educational and psychological measurement as the preferred method for validating interpretations and uses of test scores (AERA, APA, & NCME, 2014; Kane, 2013; Schilling & Hill, 2007). Scholars continue to debate the best approaches for developing and supporting validity arguments, however (for examples, see Brennan, 2013; Kane, 2007).

Purpose and Perspective
Since validation is currently discussed in terms of arguments, and since arguments are both interactive and social, the purpose of the present review is to systematically examine the structure and content of the scholarly arguments about validity and validation which appear in the peer-reviewed literature. Using theories of argumentation as a lens, researchers examine the arguments and counterarguments offered in the literatures reviewed to identify key assertions and recommendations regarding validity arguments and validation methodology. Researchers then analyze the validity arguments and evidence reported in peer reviewed journals on specific interpretations and uses of test scores to determine the degree of alignment between validation theory and practice.

Method and Data Sources
Researchers used the EBSCOhost platform to search the Education Full Text (H.W. Wilson), Education Research Complete, ERIC, and PsycINFO databases for articles published in peer-reviewed journals within the past 15 years that contain either the words “validity argument”, or the words “argument-based approach” along with “validity” or “validation”. After duplicates were removed, the search returned n = 168 articles. Per the PRISMA statement (Moher, Liberati, Tetzlaff, & Altman, 2009), researchers examined titles and abstracts to further qualify articles for the study. Articles which discuss and make recommendations regarding validity arguments qualified for inclusion in the study as methodological publications (n = 83 articles). Articles which present interpretation arguments, score-use arguments, and/or validity arguments for specific tests, along with validity evidence to support (or challenge) those arguments qualified for inclusion in the study as applied articles (n = 85 articles). Ten articles qualified in both categories, while another 10 articles were excluded from the study.

Findings and Significance
While the literatures reviewed identify a few competing concerns about validity arguments, methodological papers consistently caution against collection and reporting of validity evidence that does not directly support a thoughtful, context-specific argument. By contrast, applied papers demonstrate only partial satisfaction of methodological recommendations. The findings of this review provide a framework for the integration of methodological recommendations into an accessible framework for applied researchers.

Authors

• Matthew Ryan Lavery, South Carolina Education Oversight Committee

• Michele Carney, Boise State University

• Jonathan David Bostic, Bowling Green State University

• Jeffrey C. Shih, University of Nevada - Las Vegas

• Erin Elizabeth Krupa, Montclair State University

• Mark R. Wilson, University of California - Berkeley

• Lance Kruse, The University of Toledo