Poster #45 - Too Good to be Valid: Fraudulent Participants on Amazon’s Mechanical Turk

• In Event: Poster Session 07

Abstract

Participants gathered from online survey pools such as Amazon’s Mechanical Turk are growing in popularity among researchers. In the past 9 years Mturk’s use in published articles rose 2,117% (Aguinis, Villamore, & Ramani, 2021). As this trend increases, research to ensure the validity of these sampling techniques must be matching step (Kennedy, 2020). After examining two different studies using MTurk participants, a qualitative survey of reported parents on MTurk was found to be less valid than a quantitative study of general adults.
Two studies collected data using MTurk participants from 2019-2021. The participants of both studies were required to be over 18 years old, live in the United States, have the ability to read and write in English, and correctly answer two “catch questions” (i.e., “What is 6-4”, and “Please check the “agree” option if you’re paying attention”). The first study examined dieting behaviors in American adults. Out of 702 completed surveys, 582 delivered usable data (18% invalid surveys). Besides fully acceptable, yet uncharacteristically pervasive, metric system use for height and weight, the data displayed adequately valid responding. It did not, however, involve in-depth qualitative questions nor questions where participants were expected to answer in full sentences.
The second study sought to examine the child feeding patterns of parents on MTurk. This study gathered 68% unusable surveys (226/333). This study’s catch question only caught 13% of the invalid responders. It was the qualitative questions that identified 55% of the fraudulent surveys. It is possible that surveys designed without qualitative questions may be unintentionally publishing fraudulent data.
Participant answers to this study’s open-response questions showed a high percentage of surveys that were invalid (68%). Unusable answers to child diet questions included responses such as “Some of them are talking, others silent. Glaucon: you have given me a strange image, and they are strange prisoners” (from Plato). This response was listed on twelve separate studies with separate MTurk ID codes. Definitions from the internet and nonsense responses like “good” for each question also removed many participants. Participants who were copying and pasting information from the internet for their answers or using the same answer for numerous surveys were not found to be pattern-responding on the multiple-choice questionnaires and would have been marked as valid if the survey were only quantitative. These 184 /333 participants were scatter answering their multiple-choice answers and were only identifiable as fraudulent due to their nonsensical qualitative answers.
In summary, research using MTurk may be less valid than previously believed. While fraudulent participants may be answering multiple choice or simplistic “catch questions” accurately, qualitative surveys may be more effective in demonstrating the pervasive issues with invalid data. Further, IP addresses, spelling, metric system use, and lexicon differences, such as interpreting the word “holiday” as “vacation” supports the suggestion that fraudulent MTurk participants may be working in mass or from technology hub centers (Kennedy, 2020).

Author

• Allison Kiefner-Burmeister, University of Findlay
  Presenting Author