Poster #168 - True effect or selective reporting: Giving weight to meta-analyses measuring effectiveness of childhood obesity interventions.

• In Event: 3-053 - Poster Session 10
  In Poster Session: PS 10 Section - Prevention and Interventions

Integrative Statement

The growing body of research on weight-related interventions for youth has been summarized by several meta-analyses aiming to provide an overview of the effectiveness of interventions. Yet, the number of meta-analyses is expanding quickly and overall results differ, making a comprehensive synopsis of the literature difficult. To tackle this problem, a meta-synthesis was conducted to draw informed conclusions about the effectiveness of interventions targeting childhood obesity (Kobes, Kretschmer, Timmerman, & Schreuder, 2018). Results of the meta-synthesis showed that interventions result in small, statistically significant effects (SMD = 0.08), meaning that interventions are successful in eliciting weight loss in youth. The meta-synthesis informed about the summary of the effects, however, it is not informative as to whether there is selective reporting in the field of childhood obesity interventions. Selective reporting is the tendency to publish studies that demonstrate a statistically significant effect (Rosenthal, 1979). As a result, published papers might contain evidence unrepresentative of reality (Ioannidis, 2008). In the field of meta-science, novel techniques are applied to address this issue. In this study, one of these techniques is used to answer the following question: do meta-analyses included in the meta-synthesis measure a true effect?

P-curve analysis is a technique for determining whether a set of studies is measuring a true effect, and allows for distinguishing between significant findings that are (un)likely to be the result of selective reporting (Simonsohn, Nelson, & Simmons, 2014). With p-curve analysis, conclusions are drawn from the distribution of significant p-values from a set of studies associated with the hypothesis of interest. A right-skewed distribution suggests a true effect, a uniform distribution suggests the absence of a true effect, and a left-skewed distribution suggests selective reporting. In this study, p-values are extracted from meta-analyses included in the meta-synthesis. These p-values inform about the statistical significance of children’s BMI decrease. P-curve analysis is performed using the online p-curve app 4.06 (Simonsohn, Nelson, & Simmons, 2018).

Of the 26 meta-analyses included in the meta-synthesis, two meta-analyses could not be included in the p-curve analysis due to insufficient information concerning the p-values and test statistics. Of the remaining 24 meta-analyses, 15 significant p-values were included in the analysis. P-curve analysis showed that the 15 significant p-values present a right-skewed distribution and thus measure a true effect.

The result of the p-curve analysis shows that meta-analyses included in the recent meta-synthesis contain evidential value. This means that the statistically significant effect size measured in the meta-synthesis, which showed that interventions result in a statistically significant decrease in BMI, is likely to be a true effect. However, this effect is so small that it can be deemed trivial (Borenstein, Hedges, Higgins, & Rothstein, 2010). The results of this study demonstrate not only the value of p-curve analysis for childhood obesity intervention research, but also for meta-science in general. P-curve analysis has proven to be useful in determining whether a body of literature contains evidential value for the hypothesis of interest, and forms an appropriate tool for assessing the statistical quality of meta-analyses.

Authors

• Annita Kobes, University of Groningen
  Presenting Author

• Tina Kretschmer, University of Groningen
  Non-Presenting Author

• Greetje Timmerman, University of Groningen
  Non-Presenting Author