Retention of Participants in Online Collective Intelligence Tests Involving a Panel Design

• In Event: Online Experiments: Methods, Opportunities, and Challenges

Abstract

While online experiments offer many advantages such as convenient access to a large pool of study participants and efficient administration, recruiting and retaining participants can be frustrating for researchers. Participant dropout can also affect data quality, and limit generalizability of research findings. Yet, we know little about how exactly participants and “dropouts” differ, and what it takes to motivate participants to stay over the full duration of research. In light of these methodological and administrative issues in online experiments, we will report findings and lessons from our ongoing research that aims to test the stability of collective intelligence in interacting groups over time. Collective intelligence was measured by a group’s ability to perform well across a wide range of group tasks, using an hour-long online battery of tests we had developed. To test the temporal stability of the battery, we used a panel design, where participants recruited from a crowdsourcing company were randomly assigned to small groups to work on our battery, and were asked to come back to a follow-up session a week later to work with the same group on an equivalent version of the battery. Particularly challenging in this research were low return rates for the second session, which led to incomplete group data. In efforts to minimize participant attrition, we implemented various administration tactics during recruitment. In addition, to better understand participants who chose not to return to the follow-up study, we examined individual characteristics (e.g., personality, social intelligence, social value orientation), group-related (e.g., group affect and cognition), along with task-related variables (e.g., task awareness, intrinsic motivation pertaining to the task, behavioral pattern during the initial session). In this panel discussion, we will share recommendations for enhancing retention in longitudinal online experiments as well as insights into psychological and behavioral characteristics of the dropouts.

Authors

• Young Ji Kim, MIT Center for Collective Intelligence

• David Engel, MIT Center for Collective Intelligence

• Ishani Aggarwal, Tilburg U

• Anita Williams Woolley, Carnegie Mellon U

• Thomas W. Malone, MIT Center for Collective Intelligence