Digital and Remote Collection of Response Process Validity Evidence

• In Event: 32.054 - Thinking Through Think-Alouds: Contemporary Contributions to Validation

Abstract

The purpose of this study is to investigate the effectiveness of digitally and remotely collecting response process data as a source of empirical content validity evidence, or evidence that the item measures what it intends to measure (Boorsboom & Mellenberg, 2007).
For this study we conducted an experiment to determine the impact of different methods for collecting response process evidence. Students in each experimental group were asked to think aloud as they reasoned through a set of multiple choice questions, or to engage in what is known as a concurrent think-aloud (e.g., Kuusela & Paul, 2000). More specifically, students were asked to (a) read the question aloud, (b) verbalize their thinking as they evaluate the possible item options, and (c) describe their final answer selection, including why other options were not selected as their answer. The questions were designed to diagnose three prevalent misconceptions in probabilistic reasoning, meaning items were written explicitly to elicit specific misconceptions with specific incorrect options.
In the Spring of 2016, 75 undergraduate students participated in the study. Twenty-five students were randomly assigned to one of three experimental conditions. All three groups used a tablet or smart phone to complete the think aloud, and the audio was recorded using a free application. Group 1, the traditional administration group, completed the think aloud in a research setting with a researcher present. The second two groups completed the think aloud remotely at a time and location of their choice. Group 2 checked an iPad out from the project team and completed the assessment remotely. Group 3 completed the think aloud using their own tablet or smart phone and had no contact with the project team.
We analyzed the audio and response data to determine the extent to which remotely collecting data impacts the quality of quantity of student verbalizations when compared to student verbalizations collected using traditional, in-person methods. We investigated the impact of experimental group on five outcomes: (1) quantity of verbalizations as straightforwardly defined by the number of spoken words provided per item, (2) the accuracy of responses as defined by the total score, (3) the prevalence of misconception responses as defined by the number of incorrect options that corresponded to a misconception (misconception subscore), (4) the prevalence of misconception reasoning as defined by human rater classifications of the student reasoning and (5) the quality of the student reasoning as defined by human raters of the student verbalization according to categories of: (a) provided ample data to clearly classify student, (b) provided sufficient data to clearly classify student, or (c) provided insufficient data to clearly classify student.
Results will provide evidence to determine the extent to which remote data collection is a useful means for collecting concurrent think aloud data as a form of validity evidence. If the verbalizations are similar to those collected via in-person methods, collecting data remotely may open doors to reduce the cost and time of think aloud data collection, as well facilitate collecting samples from more diverse geographical areas.

Authors

• Laine Bradshaw, University of Georgia - Athens

• Kate Hobgood Guthrie, University of Georgia - Athens

• Meina Bian, University of Georgia