Estimating the Difference Between Published and Unpublished Effect Sizes: A Metareview

• In Event: 35.082 - Roundtable Session 14
  In Roundtable Session: 35.082-2 - Applications of Meta-Analysis in Education

Abstract

Practitioners and policymakers rely on meta-analyses to inform decision-making around the allocation of resources to individuals and organizations. It is therefore paramount to consider the validity of these results. A well-documented threat to the validity of research synthesis results is the presence of publication bias, a phenomenon where studies with large and/or statistically significant effects, relative to studies with small or null effects, are more likely to be published (Rothstein, Borenstein, & Sutton, 2005). We investigated this phenomenon empirically by reviewing meta-analyses that quantified the difference between effect sizes from published and unpublished research. All meta-analyses published in Psychological Bulletin or the Review of Educational Research were eligible for inclusion. A total of 82 meta-analyses had sufficient information to calculate an effect size between published and unpublished studies. Results indicated that published studies yielded larger effect sizes than those from unpublished studies (d ̅ = 0.18, 95% CI = [0.10, 0.25]). Moderator analyses revealed that the difference was larger in meta-analyses that included a wide range of unpublished literature. We conclude that meta-analyses should continue to include unpublished studies to mitigate the potential bias from publication status.

Author

• Joshua R. Polanin, Development Services Group