Anomalous Responses on Amazon Mechanical Turk: An Indian Perspective

• In Event: Crowdsourcing

Abstract

What can researchers do to address anomalous survey and experimental responses on Amazon Mechanical Turk (MTurk)? Much of the anomalous response problem has been traced to India, and several survey and technological techniques have been developed to detect foreign workers accessing U.S.-specific surveys. We argue that some Indian workers access and complete U.S.-specific surveys undetected, impacting data quality. We survey Indian MTurkers and find that 26% pass common, survey-based methods of detecting foreign workers. Additionally, 3% of our respondents claim to be located in the United States, implying that these respondents circumvent IP address blocking techniques designed to prevent foreign workers from accessing U.S.-specific surveys because of strong monetary incentives to do so. We show that restricting respondents to Master Workers and removing the U.S. location requirement encourages Indian MTurkers to correctly self-report their location, helping to reduce anomalous responses among U.S. respondents and to improve data quality. In keeping with this year’s conference theme, this paper approaches the MTurk quality crisis from a different perspective — that of Indian workers — in order to consider ways in which the MTurk work environment sets up incentive structures that negatively impact both workers and requesters.

Authors

• William O'Brochta, Louisiana Tech University

• Sunita A. Parikh, Washington University in St. Louis