Bias from Selective Reporting of Outcomes and Analyses in Randomized Experiments

• In Event: Advances in Evaluation Methods

Abstract

Randomized experiments with statistically significant differences are more likely to be published than those with null results. Because of this concern, some researchers selectively report outcomes or analyses even though the practice increases the risk of a type I error (incorrectly concluding that treatment is effective). Outcome reporting bias includes leaving pre­-specified outcomes unreported, introducing non­-pre­-specified outcomes without being declared as such, and under-reporting data. Selective reporting of analyses may also occur, such as exclusively reporting adjusted estimates or subgroup analyses and employing methods of handling missing data that inflate the type I error rate. This study explores the prevalence of selective reporting of analyses and outcome reporting bias in clinical trials on substance use disorders by comparing pre-specified outcomes (as reported on clinicaltrials.gov, a registry of clinical trials on a wide range of diseases and conditions) with the outcomes subsequently published from the clinical trial. The impact of reporting will be explored to determine whether it biases outcomes by producing a change in overall treatment effect size. Implications for clinical trials on substance abuse, and randomized experiments in criminology more generally, are discussed.

Authors

• Alese Wooditch, Temple University

• Lincoln Sloas, Florida Atlantic University

• Matthew Nelson, George Mason University

• Aleisha Key, Florida Atlantic University

• Xiaoyun Wu, George Mason University