Differentiating Between Types of Response Styles and Valid Responses Using Mixture and Multiprocess Item Response Theory Models

• In Event: 41.027 - Recent Advances in Modeling Response Style Bias in Rating Scale Data

Abstract

Rating scales are the most common response formats for measuring noncognitive constructs such as beliefs, attitudes, or satisfaction. They are convenient and easy to use and allow for comparing interindividual differences. However, they do not come without problems. Response styles (RS) – systematic construct-irrelevant responses – can occur due to a lack of motivation, fatigue effects, or a lack of understanding the content of an item (due to low ability of the test taker or items with poor quality), distorting the measurement by harming its validity and dimensionality. RS can be responsible for cultural and other group differences reported in educational and social sciences. Thus, they are especially a problem in low-stakes assessments, cross-cultural studies, and large-scale assessments.

Recent studies are exploring a multi-process item response theory (IRT) approach proposed by Böckenholt (2012) modeling response subprocesses to measure and correct for RS. The approach and extensions of it were successfully applied to empirical data (Khorramdel & von Davier, 2014; Plieninger & Meiser, 2014; von Davier & Khorramdel, 2013) and also evaluated using simulated data (Pokropek, Khorramdel & von Davier, under review). In one study we showed that by extending the approach to multidimensional scales (von Davier & Khorramdel, 2013; Khorramdel & von Davier, 2014), different types of RS in the data can be differentiated. In a second study we illustrated that items can have loadings on both a general factor (RS) and a specific factor (trait) by applying a bifactor IRT model to the response subprocesses. We also illustrated that a bifactor model can identify to what extent data are biased by RS. Using simulated data (Pokropek, Khorramdel, & von Davier, under review), we were able to show in which situations, in terms of types of RS and amount of RS bias in the data, the multi-process IRT approach can successfully detect RS. We also showed that a two-dimensional extension of the approach can disentangle different types of RS. In the same study, we successfully validated the approach on empirical data and showed that RS are more likely to occur in groups with a higher number of omitted responses (in a previous cognitive assessment), higher scores of overclaiming, and low reading ability than in groups with a low number of omitted responses, low overclaiming scores, and high reading ability.

The findings of our studies show that the measurement of RS is not straightforward but can vary for different subset of items and groups of respondents. Thus, in ongoing studies using multigroup and mixture IRT models, we examine different approaches to better differentiate between groups of respondents who give valid responses versus RS. We are also examining the relation between RS and other variables (e.g., timing information) that might be helpful in identifying meaningful groups in regard to RS. The goal of our research is to provide tools that will support comparability in the assessment of the constructs of interest, meaningful scores, and unbiased measurement of group differences by correcting test scores for RS.

Authors

• Lale Khorramdel, ETS

• Matthias Von Davier, National Board of Medical Examiners

• Artur Pokropek, Polish Academy of Science