A Hierarchical Framework for Validation

• In Event: AERA Division D Robert L. Linn Distinguished Award Address by 2020 Awardee Michael T. Kane

Abstract

Assessments assume an interpretation and use for their results, and validation requires an evaluation of how well the scores support these interpretations and uses. Use-specific models of validity assume a particular use for the scores (e.g., admissions, certification of achievement, diagnosis), and an interpretation that supports the intended use; these specific models can be fairly easy to validate, because the model specifies kinds of evidence needed for validation. General, unified conceptions of validity are designed to provide a general framework for validation and to apply to a range of interpretations and uses. A general, argument-based approach to validation involves the specification of the proposed interpretation and use of the scores as an interpretation/use argument (IUA), and then, an evaluation of the completeness, coherence, and plausibility of the IUA. This general framework can be difficult to apply, if we think of developing the IUA from scratch for each validation. A hierarchical, argument-based framework for validation adds layers of more specific models of IUAs that provide more detailed guidance and standards for validation. The presentation will outline a number of more detailed, use-specific models such as value-added models, monitoring vs. accountability models, licensure vs. employment models, and causal models.

Michael T. Kane has been the Messick Chair in Validity at the Educational Testing Service since 2009. He served as Director of Research at the National Conference of Bar Examiners from 2001 to 2009, and as a professor in the School of Education at the University of Wisconsin from 1991 to 2001. Prior to 1991, Dr. Kane served as VP for research and development and as a senior research scientist at American College Testing (ACT) in Iowa City, as Director of Test Development at the National League for Nursing in New York, as a professor of education at SUNY, Stony Brook, and as Director of Placement and Proficiency Testing at the University of Illinois, Urbana-Champaign. His main research interests are validity theory and practice, generalizability theory, licensure and certification testing, and standard setting. Dr. Kane holds a Ph.D. in education and an M.S. in statistics from Stanford University, and a B.S. and M.A. in physics from Manhattan College and SUNY, Stony Brook, respectively.

Author

• Michael T. Kane, ETS