Stop & Think: Can Norm-Related Prompts Influence Online Commenters’ Behavior?

• In Event: Correction! Can We Mitigate Misperceptions and Incivility?

Abstract

Can prompts from the website/platform cause users to reconsider their behavior? And what kind of prompts can motivate people to reduce the level of incivility or intolerance in their comments? Enforcing a “breather” and giving people a chance to reconsider whether they want to share or say something online is one of the least censorious, intrusive and resource-intensive interventions a publisher or platform can make. This can help attack the vexing problems of user-generated content online like incivility, intolerant discourse or false and/or defamatory content. Experiments prompting users to rethink whether to share or produce certain content online have yielded mixed results, suggesting that the message type as well as recipient characteristics matter, and sometimes unintended effects may occur (Clayton et al., 2020, Epstein et al., 2021, Freeze et al., 2021, Penycook et al., 2021).

Conducting an experiment on some of the major news websites in Romania, we test what kind of messages can motivate online commenters to “stop and think” twice before posting and even to edit their comments and examine whether such an intervention changes the content of comments and reduces the need for comment moderation. Specifically, we test the impact of showing commenters certain pop-up warning messages when they are about to post a comment, messages that allow them to review and edit their comment or submit it as is.

The first iteration of the experiment took place in two phases in late 2015 - early 2016, In phase 1, we tested the impact of one generic warning message applied to a random sample of half of situations when commenters tried to submit a comment on the websites of major Romanian mainstream media outlets. In phase 2, we tested the impact of five different types of messages - with different appeal types related to descriptive or injunctive norms (Bicchieri, 2016), targeting distinct motivations - randomly assigned to every single case when a commenter attempted to submit a comment (January - April 2016).

The analysis of a corpus of over 250,000 comments indicated that a very small share of comments were revisited and yielded unclear results as to whether edited comments were ultimately more or less civil than the unedited drafts. Within person effects could not be computed because of the very small share of users who commented repeatedly and differences of different types of users could only be restricted to their characteristics as commenters not other theoretically relevant attitudinal or socio demographic characteristics. Moreover, it was unclear if warning messages were always shown, visible and seen by the users.

Our current research aims to build on these findings and provide fresh tests of the hypotheses regarding civility prompts and their contingencies. We investigate further whether, when and on whom prompts work and the comparative effectiveness of various appeal types. To investigate further what kinds of prompts may be most effective among what kinds of users, as well as to ensure that warning messages are delivered and cannot be so easily overlooked as they are on a website, we conduct an experiment on an online panel of respondents, simulating a comment section environment. We then conduct the experiment on a non-Romanian sample on Mechanical Turk. Finally, informed by intermediary findings we implement a modified version of the original comment-section level experiment on the same Romanian media website.



Bibliography

Bicchieri, C. (2016). Norms in the wild: How to diagnose, measure, and change social norms. Oxford University Press.

Freeze, M., Baumgartner, M., Bruno, P., Gunderson, J. R., Olin, J., Ross, M. Q., & Szafran, J. (2021). Fake claims of fake news: Political misinformation, warnings, and the tainted truth effect. Political behavior, 43(4), 1433-1465.

Clayton, K., Blair, S., Busam, J. A., Forstner, S., Glance, J., Green, G., & Nyhan, B. (2020). Real solutions for fake news? Measuring the effectiveness of general warnings and fact-check tags in reducing belief in false stories on social media. Political Behavior, 42(4), 1073-1095.

Epstein, Z., Berinsky, A.J., Cole, R., Gully, A., Pennycook, G., et al. 2021. "Developing an accuracy-prompt toolkit to reduce COVID-19 misinformation online." Harvard Kennedy School Misinformation Review.

Pennycook, G., Epstein, Z., Mosleh, M. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595 (2021). ttps://doi.org/10.1038/s41586-021-03344-2

Authors

• Tania Chilin, Median Research Centre

• Raluca Toma, Fundatia MRC - Median Research Centre

• Marina Popescu, Median Research Centre