The Effect of Drag-and-Drop Item Type Design on Test-Takers' Performance and Strategy Use

• In Event: 79.017 - Technology-Enhanced Items in Educational Assessments: Challenges, Findings, and Implications

Abstract

The drag-and-drop item type is one of the most commonly used Technologically Enhanced Item type in digitally-based assessments. Using this item type requires that test-takers create correct responses by selecting from the sources provided then dragging that selection and dropping it into a target rather than recognizing the correct response from a list presented with the item. This feature of the drag-and-drop item type allows us to understand not only whether a test-taker knows the answer or not, but also the strategies a test-taker uses to get to the answer in the form of the process data.
Currently, decisions regarding design and use of drag-and-drop items are based on experts’ suggestions (e.g., Parshall, Depascale, & Skinner, 2011; Parshall & Cadle, 2014; Parshall & Brunner, 2017), design constraints, and client requests rather than empirical research. Our goal is to contribute to the transition from technology-centered approach to assessment development in which the drag-and-drop item type is used as a way to engage with technology, to a cognition-centered approach in which the drag-and-drop item type is used by considering the cognitive affordances of this item type.
To understand the effect of drag-and-drop item design on test-takers’ strategy use to get to the answer (i.e., dragging the sources in a sequential manner into the targets, irrespective of the order of the targets – source-focused strategy– vs. dragging the sources into the targets sequentially, irrespective of the order of the sources – target-focused strategy) and on the test-takers’ performance, we investigate the following research questions:
(i) Does the item design of a drag-and-drop item have any effect on test-takers’ strategy use to give a response?
(ii) Does the design of a drag-and-drop item have any effect on test-takers’ performance?
To this end, we conducted an experiments on MTurk by using 10 drag-and-drop mathematics items. The experimental conditions have the following manipulations:
• Presenting first sources and then targets;
• Presenting first targets and then sources;
• Swapping the content of the sources and targets;
• Manipulating the item stem in a way that the presented information cannot be chunked to give an answer;
• Manipulating the item stem in a way that a test-taker has to look at the targets to understand the instruction.
The analyses of process data (i.e., action sequences and time spent on items) and scores will be presented. Overall, the findings are expected to contribute to a wide variety of assessment programs in making empirically informed decisions when designing drag-and-drop items.

Authors

• Burcu Arslan, Educational Testing Service

• Yang Jiang, Educational Testing Service

• Tao Gong, ETS

• Madeleine Keehner, Educational Testing Service