Using Think-Aloud Protocols to Investigate Item Wording Effects

• In Event: 32.054 - Thinking Through Think-Alouds: Contemporary Contributions to Validation

Abstract

In this presentation, we will discuss two studies in which we used think alouds to investigate the effects of changes in the wording of affective items. In Study 1 we experimentally manipulated three item wording characteristics: vague versus specific wording (changing the term “almost always” to “95% of the time”); phrasing items as question rather than statements, and labeling each response option versus labeling only the endpoints. In the quantitative phase of the study, we analyzed differences in responses across the three manipulated conditions. In the think aloud phase of the study, we provided respondents with two scales: one scale in which item characteristics had been manipulated and a separate scale in which they had not. Respondents were asked to think aloud as they answered each item. We then transcribed their responses and looked for themes related to the item characteristics. Specifically, we were interested in whether respondents remarked on the manipulated characteristics and whether these characteristics seemed to affect their responses.
Procedures for Study 2 mirrored those for Study 1, but only the vague versus specific wording and the statement versus question manipulations were included. To investigate the generality of the results from Study 1, the manipulations were applied to different scales. As in Study 1, we chose eight respondents for a think aloud study, and these respondents were provided with one manipulated and one non-manipulated scale.
We briefly present some illustrative results. The vague versus specific wording manipulation seemed to affect the answers of at least some of the respondents. For example, the item “Beliefs should be revised in response to new information or evidence” was manipulated to read “Beliefs should always be revised in response to new information or evidence.” One respondent stated “I would say disagree slightly because … your belief is your belief, so you can revise it, that’s your opinion, it doesn’t always have to be revised (italics added). This respondent elaborated on the issue by contrasting how she would have answered the question if the word “always” had been changed to “often”: “Often is different than always, because like often is every once in a while whereas like always is always, like definitely has to happen, like beliefs should always be revised, because it says like always.”
The partially labeled versus fully labeled scales also elicited some relevant comments. As one respondent stated “It [the partially labeled response scale] made you like think more about which one you wanted to choose, and it honestly like makes you like not choose any of them, like you just choose 1 or 7 [the labeled options] because you see like exactly what it means. Like even though you can like ration in your head like which ones the other means, the other ones means, you just kind of ignore them because it doesn’t have anything there to explain what they are.” Another respondent stated “I thought of the words; I didn’t pay attention to the numbers.”

Author

• Deborah L. Bandalos, James Madison University