Blinded by the Light: P-Value Misreporting and Excess P-Values Just Below .05 in Communication Science

• In Event: Methodological Concerns in Mass Communication Research (TOP PAPER ENTRY)

Abstract

Publication pressure and/or verification bias motivate researchers to report statistical relationships at the p<0.05 (significance) level. These motivations may yield (1) (un)intentional p-value misreporting, and (2) excess prevalence of p-values just below .05. To examine whether such distortions occur in Communication Science, we analyzed 5,834 test statistics extracted from 692 recent (2010-2012) papers from 25 ISI-rated Communication Science journals, as well as 19,830 test statistics extracted from 28 Social Psychology ISI journals.
In Communication Science journals, 6.8% of p-values were misreported; most were erroneously downsized. 0.9% stated p<.05 while in fact p>.05. P-values between .055 and .01 were up to 166% more prevalent than expected based on p-value probability density distributions. Stronger discrepancies occurred just below .05. Results were strikingly similar for Social Psychology, although this field was less biased toward downsizing flawed p-values.
We conclude that the Communication Science knowledge base is distorted, and suggest solutions to this problem.

Authors

• Ivar Vermeulen, VU U Amsterdam

• Camiel J. Beukeboom, VU U Amsterdam

• Anika Batenburg, Radboud University

• Dimo Stoyanov, VU U Amsterdam

• Bob Robbert Nicolai van de Velde, VU U Amsterdam

• Dirk Oegema, VU U Amsterdam