Validating Massively Open Online Social Experiments With Volunteer Science

• In Event: Online Experiments: Methods, Opportunities, and Challenges

Abstract

Traditional laboratory experiments in social behavior have been based on recruiting college students to participate in person in exchange for payment. This model is limited by its inability to scale to group experiments, the costs involved, and specificity of its subject pool. To address these issues, we have built and tested an online laboratory as an alternative approach to social and behavioral experimentation called Volunteer Science (www.volunteerscience.com). In this presentation, we discuss our alternative model for massively open online social experiments which utilizes web technologies to recruit, manage, and implement large-scale experiments with online participants. Following recent research investigating the use of web-based technologies to perform social experiments, we replicate several canonical effects across various sub-fields to test the validity of the platform. We test the Stroop and Flanker effects which elicit low-latency perceptual delays to demonstrate our ability to induce and detect effects at the resolution of 10’s of milliseconds. We implement a battery of cognitive biases tasks from behavioral economics to test our users’ reasoning and compare them to known samples. Third, we implement a Big Five personality scale to demonstrate our ability to capture multi-dimensional latent traits through surveys. Finally, we implement an online version of a classic network experiment to demonstrate the platform’s ability to manage network and group experiments. The results of these replications are largely confirmatory as most users participated in expected ways and we replicate the cognitive, personality, network, and most heuristic effects. However, users diverge in several distinct ways. Compared to students in laboratories, online participants, both Amazon Turk workers and volunteers, appear to be less susceptible to some cognitive biases and are more quantitatively circumspect. And, online volunteers are less likely to consent to research, introducing a new layer of non-participation bias.

Authors

• Jason Radford, U of Chicago

• Brian Christopher Keegan, Northeastern U

• Katherine Ognyanova, Northeastern University

• Brooke Foucault Welles, Northeastern U

• Jefferson Hoye, Jefferson Hoye LLC

• Ceyhun Karbeyaz, Northeastern U

• Waleed Meleis, Northeastern U

• David Lazer, Northeastern U