Metasynthesis of Studies in “When Validity Theory Meets Validation Practices” With an Eye Toward Comparing and Contrasting the Seven Research Syntheses

• In Event: When Validity Theory Meets Validation Practice: Research Syntheses of Validity Evidence Reported in Seven Areas

Abstract

Objectives: This paper aims to summarize and present in detail the common trends as well as the uniquenesses found across these seven validity studies. An attempt is made to provide insight into where the research on validity presently stands, how it has changed from its inception, and where it is heading across a broad range of disciplines and journals in the educational, psychosocial, and health sciences domains. In terms of our meta-synthesis, emphasis is placed on the improvements and the benefits that validation-oriented research has for education and the importance of engaging in it to appropriately use and interpret educational and psychological tests and measures.

Methods/Data Source: The seven research syntheses have common categories based on the Test Standards (AERA, APA, & NCME, 1999). In particular, each synthesis provided a numerical summary for the five sources of validity evidence in the Standards: a) content-related, b) response processes, c) internal structure, d) associations with other variables, and e) consequences. Data analyses were conducted comparing the seven syntheses on these five sources of evidence, as well as on the percentage of papers citing or integrating validity theory or frameworks (e.g., Kane, 2006; Messick, 1989, the Test Standards). For those syntheses that included a temporal comparison, we also compared trends over time. Our meta-synthesis, however, did not focus exclusively on numerical analysis but also compared and contrasted the more qualitative features of the seven syntheses.

Results & Conclusions: Overall, the seven syntheses show mixed results in terms of the trends that have taken place in validation research over time. Although there is a wider acknowledgement of the importance of validity and an increase of researchers trying to empirically ground the usefulness and appropriateness of the conclusions derived from their tests/measures, there is still some way to go. The use of the Standards is practically non-existent and, although there is little recognition of the modern/unitary view of validity, the fact of the matter is that most validation processes are still firmly grounded in early 20th century conceptions. Reliability indices and other forms of internal-structure statistical analyses are, by far, the most widely used forms of evidence that are presented (sometimes incorrectly) as “validity evidence”. Construct-related validity evidence (through relationships and comparisons with other variables) is the second most widely reported, although there seems to be confusion with regards to terminology (e.g., misunderstandings on discriminant VS discriminative, criterion-related validity evidence being presented as predictive-validity, etc.). Content-related validity evidence has received very little attention in validation research. Finally, validity evidence based on response processes and consequences has been virtually ignored. It is important, but not surprising, to note that the measurement focused journal Educational and Psychological Measurement was the most aligned, compared to the content domain focused journals such Journal of Educational Psychology¸ with the contemporary views of validity and test validation.

Authors

• Yan Liu, The University of British Columbia

• Juliette Lyons-Thomas, The University of British Columbia

• Oscar L. Olvera, The University of British Columbia

• Bruno D. Zumbo, The University of British Columbia