Testing for Funnel Plot Asymmetry of Standardized Mean Differences

• In Event: Reconceptualizing Practical Meta-Analysis in Education

Abstract

A critical threat to the validity of findings from research syntheses is outcome reporting bias, which occurs when the sample of primary study results identified for synthesis are not representative of the full scope of relevant research on the topic of the synthesis. One mechanism that can produce outcome reporting bias is a preference (among editors, reviewers, and authors) for publishing statistically significant findings, which leads to under-representation of statistically non-significant results in the published record (Rothstein, Sutton, & Borenstein, 2005). Many research synthesis projects now seek to detect the presence of outcome reporting biases. One class of techniques for doing so focuses on the funnel plot, a scatter plot of effect size estimates against a measure of the precision of the estimates (Light & Pillemer, 1984).
Methods for detecting outcome reporting bias from funnel plots are premised on the assumption that the scatter plot will show a symmetric, funnel-shaped distribution of points in the absence of publication bias, but will appear asymmetric if there is selective reporting of statistically non-significant results. Formal statistical tests of funnel-plot asymmetry include Begg and Mazumdar’s (1994) rank-correlation test, the Trim-and-Fill technique (Duval & Tweedie, 2000), Egger’s regression test (Egger, Smith, Schneider, & Minder, 1997), and the more recent variants of Egger regression known as PET and PEESE (Stanley & Doucouliagos, 2014). However, it has been recognized that the assumption of funnel plot symmetry does not always hold for odds ratios, a common effect size measure used in meta-analyses of clinical medicine, and modified tests have been proposed that rectify the problem for odds ratio estimates (Moreno et al., 2009; Peters, Sutton, Jones, Abrams, & Rushton, 2006).
The purpose of the present paper is to demonstrate that conceptually similar problems occur with standardized mean differences (SMDs), a ubiquitous effect size in educational and psychological research. Specifically, the standard error of the SMD is correlated with the estimate itself, leading to asymmetry whenever the average true effect is non-zero. This induced correlation can be avoided by using the scaled standard error of the unstandardized effect (i.e., the standard error of the SMD numerator, divided by the SMD denominator) as the measure of precision in existing tests of funnel plot asymmetry.
To verify these observations, we conducted a Monte Carlo simulation to estimate operating characteristics of conventional and modified funnel plot asymmetry tests. We simulated SMD effect size estimates from two-group experiments, using an empirical distribution of sample sizes, population average effects varying from 0 to 1.0 SMD, varying degrees of between-study heterogeneity, and varying degrees of publication bias. Simulation results indicated that the conventional tests have inflated Type-I error, while the modified tests maintain correct Type-I error rates. The full paper will also examine the relative power of the modified tests and the accuracy of adjusted estimates of population average effects. We conclude that tests of funnel plot asymmetry should be conducted using modified measures of precision that are uncorrelated with the effect size estimates.

Authors

• Melissa Angelina Rodgers, The University of Texas - Austin

• James Eric Pustejovsky, The University of Texas - Austin