Identification of Response Styles in Attitude Measurement Using a Multiprocess Mixture Item Response Theory Model

• In Event: 41.027 - Recent Advances in Modeling Response Style Bias in Rating Scale Data

Abstract

The vulnerability of attitude measures to response style (a tendency of respondents to prefer certain scale points regardless of item content) is well documented in the literature (e.g., Bolt & Johnson, 2009; Kam & Meyer, 2015; van Vaerenberg & Thomas, 2013; von Davier & Khorramdel, 2013). Response style can inflate or attenuate individuals’ ratings, thereby contributing to measurement error of the construct under study. Analytically, response style poses unique problems to naïve IRT models. It has been shown that because item and person estimates are codependent, a single biased response necessarily contaminates the estimates of other unbiased responses (Jin & Wang, 2014). The recent proposition of multiprocess IRT models (Böckenholt, 2012) separates the directionality of the response from the intensity of the response, which allows deeper analysis of response style. We propose that modeling latent, rather than manifest, groups displaying response style will prove useful, due to weak associations between response style and demographic covariates.

In study 1, we conducted a simulation to quantify the impact response style had on noncompensatory IRT models. We simulated 2,000 Likert-type responses to 25 unidimensional items with specified item slopes, difficulties, and person-abilities using Samejima’s (1968) graded response model. In study 2, we used archival survey data from hospital employees, as well as simulation, to study the usefulness of multiprocess mixture models. A total of 1,093 de-identified archival responses to 9 unidimensional job attitude items from employees in a VA hospital collected in 2014 were analyzed.

In study 1, we found that even a small percentage of biased responses led to mis-estimation of ability of unbiased responses. The impact was greater for biases towards the extremes of the response scale rather than towards the middle. When just 5% of cases were biased, 4% of ability estimates for unbiased cases flipped from above 0 to below 0. In study 2, we found the multiprocess mixture IRT approach useful in separating directionality and intensity. We also found distinct latent classes with specific compositions of respondents and specific tendencies to use certain response scale points. We conducted another simulation which showed good ability to identify latent classes. This method preserves information that would be lost if traditional methods were used which reductively correct for response style based on manifest demographics. This work advances the theoretical and methodological efforts towards more accurate measurement of latent constructs.

Authors

• Ryan Derickson, U.S. Department of Veteran Affairs

• Lihshing Leigh Wang, University of Cincinnati

• Jiaqi Zhang, SKT Education Group