Assessing the Effect of Outcome Reporting Bias on Weisburd's Paradox

• In Event: Design Challenges in Quantitative Research

Abstract

The Weisburd Paradox refers to an anomaly that challenges conventional assumptions of experimental research design. It postulates that, in practice, randomized experiments with fewer participants are better able to detect treatment effects. Gelman (2016) posits that the Weisburd Paradox is a result of Type M (magnitude) error and outcome reporting bias in which outcomes in smaller sample studies with statistically insignificant findings go unreported. This study relies on a sample of clinical trials on substance use reported on clinicaltrials.gov, a registry that mandates the registration of protocols and pre-specified outcomes for randomized experiments on a wide range of diseases and conditions. The purpose of this study is to reassess the existence of the Weisburd Paradox, after controlling for (suspected) outcome reporting bias. The findings from this study will provide a more nuanced understanding of the relationship between sample size and effect size in experiments, and will address whether outcome reporting bias may contribute to the Weisburd Paradox.

Authors

• Matthew Nelson, George Mason University

• Alese Wooditch, Temple University

• Lincoln Sloas, Florida Atlantic University

• Aleisha Key, Florida Atlantic University

• Xiaoyun Wu, George Mason University