Research Synthesis of Validity Evidence in Four Counseling Journals

• In Event: When Validity Theory Meets Validation Practice: Research Syntheses of Validity Evidence Reported in Seven Areas

Abstract

Objectives
To investigate the reporting of validity evidence of measures used in counseling, we conducted a review on papers published in four major counseling journals in 2009 and 2010.

Perspectives
In counseling, measures are used for assessing career interests, mental health functioning, and the effectiveness of counseling interventions, etc. In a review of papers reporting measurement properties of measures published in the Journal of Counseling Psychology in 1967, 1977, and 1987, Meier and Davis (1990) found that the rates of reporting validity evidence were very low. Measurement validity remains an important area in counseling (Bolt & Rounds, 2000) and is a fundamental issue in and evaluating measures (AERA, APA, NCME, 1999). The modern, unitary view of validity contends that validity is “an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores” (Messick, 1989, p. 13).

Methods and Data Sources
We reviewed validity papers published in 2009 and 2010 in four major counseling journals. The four journals included the 1) Journal of Counseling Psychology, 2) Counseling Psychologists, 3) Canadian Journal of Counselling and Psychotherapy, and 4) Journal of Counseling and Development. We searched both the titles and abstracts using “validity” and “validation” as the keywords. Papers reporting empirical evidence on measurement validity were included. Validity evidence reporting was recorded using a coding sheet developed by the authors.

Results
A total of 16 papers were included in this analysis. The percentage of the sources of validity reported in the four counseling journals included: Face (6.3%), content (18.8%), construct (43.8%), predictive (0%), concurrent (13%), discriminant (25%), convergent (38%), and internal consistency (56%). The unitary view of validity, response processes, and consequences were nonexistent in the validity papers published in the four major counseling journals.

Scientific Significance
Our review of the four major counseling journals showed that validity information is not reported using the modern view of validity as a guide. The unitary view of validity, citation of theoretical papers in validity theory, and the distinction between validity and validation rarely existed in the papers we reviewed. Consequences and response processes were nonexistent. More work is needed to narrow the gap between validity theory and validation research in counseling.

Authors

• Eric K. H. Chan, The University of British Columbia

• Bruno D. Zumbo, The University of British Columbia

• Roya Vojdanijahromi, The University of British Columbia