Research Synthesis of Validity Evidence in Papers Published in a Journal on Patient-Reported Outcomes

• In Event: When Validity Theory Meets Validation Practice: Research Syntheses of Validity Evidence Reported in Seven Areas

Abstract

Introduction
Patient-reported outcome (PRO) and related psychometric instruments are increasingly used in health outcomes research. Measurement validity is a fundamental consideration in developing and evaluating measures (e.g., AERA, APA, & NCME, 1999). Validity is not only about whether a measure measures what it purports to measure. Modern view of validity contends that validity is about the inferences, claims, or decisions that we make based on measurement scores. Construct validity is the focus of validity. In addition, validity does not exist as distinct types and validation should not be a piecemeal activity. This is referred to as the unitary view of validity (Messick, 1989). Validation is also an ongoing process (because changes in our societal value, empirical knowledge, etc. occur over time) in which we accumulate and synthesize validity evidence to support the inferences, interpretations, claims, actions, or decisions we make. And finally, consequences of measurement are part of the validation process.

Objectives
With an eye toward investigating the validity evidence and refining and improving the methodology of validation practice in outcomes research, we systematically reviewed the reporting of validation practice in papers published between 1998 and 2010 in Value in Health, one of the major journals in the area of PRO.

Methods
A systematic search using the official website of the journal was conducted in December 2010. We searched for papers published since the journal’s inception (January 1998) to December 2010. We searched both the titles and abstracts. The searches were independently conducted by two of the authors. Keywords used in the search included “development OR measurement OR psychometric OR psychometrics OR valid OR validation OR validity.” Papers reporting empirical results on the validity of PRO, HRQoL, and related psychometric instruments were included. Econometric, opinion, and conference papers were excluded. A coding sheet was developed and each included article was double-coded independently by two of the authors.

Results
A total of 93 papers were included (approximately 50% were disease-specific measures). The percentage of reports of the broad categories of validity evidence were: construct (39.8%), discriminant (28.0%), convergent (26.9%), content (17.2%), known group (17.2%), concurrent (14.0%), predictive (5.4%), response processes (4.3%), and consequences (1.1%). A paper may report more than one type of validity evidence. In addition, 95.7% of the papers referred to the validity of the instrument, rather than the modern view of validity of the inferences from the scores.

Discussion and Scholarly Significance
Validity evidence and validation practice in papers published in Value in Health are not routinely presented from a modern validity perspective; that is, consider a unitary view of validity. Certain sources of validity evidence (e.g., consequences and response processes) are essentially ignored.

Authors

• Eric K. H. Chan, The University of British Columbia

• Bruno D. Zumbo, The University of British Columbia

• Ira Darmawanti, State University of Surabaya

• Olievia Prabandini Mulyana, State University of Surabaya