Quantified "Member Checking" as an Assessment of Item-Level Validity

• In Event: Item Development

Abstract

For a measure to be valid, its items must measure the underlying construct of interest. However, if a participant fundamentally understands the goal of an item or provided response scale differently than the developers of the instrument, it is impossible for the meaning (construct) that the instrument developer intends to capture to be reflected fully in the score derived from that response. This paper demonstrates a new approach to modeling such potential misalignment and quantifying its impact on item validity using structured residuals. Results show that alignment items accounted for a non-negligible portion of error variance, supporting the notion that differences between participant and developer framing of content can significantly impact questionnaire response accuracy and validity.

Authors

• David F. Feldon, Utah State University

• Kaylee Litson, Utah State University