Validity evidence presented in the Journal of Educational Psychology across two temporal periods

• In Event: When Validity Theory Meets Validation Practice: Research Syntheses of Validity Evidence Reported in Seven Areas

Abstract

Purposes
This study examined validity evidence presented in articles published in the Journal of Educational Psychology (JEP) within the 1950s and the 2000s. The aim of this study was to explore a) whether validation practice has changed over time and b) the degree to which validation practice in recent articles reflects modern validity theory.

Perspectives
The 1950s and the 2000s were explored because they represent two very different times in validity theory. In the 1950s, the predominant view of validity was on types of validity (American Psychological Association [APA], 1954). However, it was also a time when long held beliefs about validity were questioned and modern concepts were proposed. In contrast, articles published in the 2000s reflect current validation practice. The modern view of validity refers to five types of evidence (American Educational Research Association [AERA], APA, and National Council on Measurement in Education [NCME], 1999). More specifically, validity is the “degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests” (AERA, APA, & NCME, 1999, p. 9).

Methods and Data Source
To obtain data for this study, a search of articles published in JEP in the two decades was conducted through PsychARTICLES database. Articles that contained the terms validity or validation were included (n = 16 for 1950s, n = 19 for 2000s). The articles were then analyzed in relation to five categories: a) characterization of validity (e.g., unitary view), b) sources of evidence, c) comparison variables (e.g., criterion-related, convergent, or discriminant variables), d) coefficient alpha levels, and e) number of sources presented.

Results and Conclusions
Because of the conceptualization of validity in the 1950s, it is understandable that many of the modern views were not presented in articles; however, many of these views were not presented in modern articles either (e.g., no articles from the 2000s provided evidence based on response processes or consequences). Furthermore, validity was overwhelmingly presented in similar ways in both decades. For example, very little change was evident in the reported conceptualization of validity, the distributions of the most reported sources of validity, the lack of justification for the choice of comparison variables, and the number of sources reported in each article.
There were some exceptions. The two main encouraging changes were the great increase in articles published in the 2000s providing some form of construct related evidence and evidence of internal structure. This suggests that some aspects of modern validity theory have entered practice. Overall, however, there appears to be a considerable discrepancy between validity theory and validation practice in articles recently published in JEP. These results will be elaborated for AERA.


Significance
This study highlights that more must be done to reduce the gap between validity theory and validation practice. To start, journals like JEP should increase their expectations of what and how validation practice is reported in validation articles. In turn, this will encourage researchers to undertake validation practice that corresponds with modern validity theory.

Authors

• Rebecca J. Collie, The University of British Columbia

• Bruno D. Zumbo, The University of British Columbia