Investigation of Option Position Effects in Four-Choice Multiple-Choice Items With Generalized Linear Mixed Model

• In Event: Using Mixed Models to Investigate Tendency, Process, and Typology in Item Responding

Abstract

1. Purpose
This paper aimed to investigate whether option positions influence option selections, and how option position effects vary with examinees’ ability using generalized linear mixed model (GLMM). The present work aimed to explore option position effects in observational data from large scale standardized assessment, and to investigate the presence of systematic position effects when the assessment has been key-balanced, in which the correct answers are reassigned to different positions under certain rules, one of them being even distribution among the response options (Bar-Hillel & Attali, 2002).
2. Perspectives
Multiple-choice (MC) items are prevalent in large scale educational testing due to their easiness, speed, and objectivity in scoring by both humans and machines (Baghaei & Amrahi, 2011). However, there has been criticisms that construct-irrelevant factors could contaminate what the test is designed to measure (Hohensinn & Baghaei, 2017). One of such factors is option position, which are “strong and systematic position effects in the behaviour of both test takers and test makers” (Attali & Bar-Hillel, 2003).
Previous experimental studies have shown that 70 to 80 percent of the test makers and test takers chose to hide or guess the answers in the middle positions in 4-choice MC items (Berg & Rapaport, 1954; Attali & Bar-Hillel, 2003). Nearly 30 percent more examinees chose the distractors in the middle positions in comparison to the same distractors presented in the edge positions (Attali & Bar-Hillel, 2003). However, experimental manipulations may not always be viable in investigating option position effects in real assessment data.
3. Methods
The data of the 2015 TIMSS USA Grade 4 Mathematic were used. A total of 87 4-choice MC items were retained from 14 booklets. Each item appeared twice in two different booklets. There was a total of 10,029 students participated in the assessment. In the global (over all booklets) level, the observed keys were close to be perfectly balanced, while in the local, booklet level, it appeared to resemble a key-balanced approach, except for booklet 11, in which there was no D-keyed item.
GLMM was used in modelling the relationships between option selection and option positions while controlling for examinees’ ability and keying (difficulty).
4. Results
Our results showed a statistically significant position effects for positions B and D. The odds for choosing B were 1.21 times the average odds, while the odds for choosing D were .75 times of the average odds. The statistically significant interaction effects between option positions and ability showed that the odds for choosing A and B increased as the ability of an examinee decreased.
5. Significance
Unlike previous experimental method, this study showed the use of GLMM to investigate option position effects, while controlling for ability and difficulty, with observational data that reflected the real response tendency in real testing contexts. Option position effects were found to be present, albeit the seemingly key-balancing approach adopted, and interact with ability in explaining option selections, showing stronger position effects in options A and B among weaker examinees.

Authors

• Lok Heng Chau, The University of British Columbia

• Minjeong Park, The University of British Columbia

• Shun-Fu Hu, The University of British Columbia