Banning the Bots: Survey data collection in the era of generative AI

• In Event: Implications of AI for Qualitative, Quantitative, and Computational Sociological Research

Abstract

Online surveys using nonprobability sampling can be a highly effective tool for researchers seeking to collect data on a budget, particularly when studying hard-to-reach populations. We use data from a pilot survey advertised on Meta and administered via Qualtrics to evaluate the effectiveness of common bot-detection strategies and other tools to protect data quality (n = 383). Despite best efforts to prevent data quality issues, 90% of responses were flagged on one or more issues. This study makes three contributions to the growing literature on the quality of online survey data. First, we examine the relationship between incentive distribution and survey response behavior, providing evidence of whether survey links are actively monitored and distributed by survey or click farms. Second, we evaluate the comparative effectiveness of traditional, proprietary, and logic-based detection strategies. Third, we provide recommendations for researchers surveying hard-to-reach populations online, where the balance between rigorous fraud-prevention strategies and participant access and protection is particularly critical. Together, we help advance a more comprehensive methodological framework for understanding and addressing participant fraud in online survey research in the era of generative AI.

Authors

• Laura DeMarco, North Carolina State University

• Alexis Jones, North Carolina State University

• Sadé Lindsay, Cornell University