Multilevel Meta-Analysis of Standardized Single-Case Experimental Data: Bias Corrections in Estimating Variance Components

• In Event: 51.029 - Handling Methodological Complexities in Effect Size Calculation and Meta-Analysis of Single-Case Experimental Design Data

Abstract

Previous simulation studies evaluating the use of multilevel models to combine single-case experimental data (level 1) over cases (level 2) and studies (level 3) have consistently shown that recovery of variance components for the higher levels of the multilevel model (i.e., between-case variance and between-study variance) is rather poor.
In this poster, we report on a simulation study that was designed to assess estimation of higher-level residuals’ variance components in the meta-analysis of standardized raw data and standardized effect sizes from multiple baseline designs. The research question is whether and to what degree the results can be improved by applying different adjustment models. For standardized raw data, the results of ‘traditional’ multilevel meta-analysis using restricted maximum likelihood without any correction were compared to those obtained when applying Hedges’ bias correction, or when constraining the level-1 residual’s variance to one. For standardized effect sizes, the standard maximum likelihood approach without any correction was compared to the application of Hedges’ bias correction. We primarily looked at the variance component estimations for both the immediate treatment effect and the effect on trend to check to what extent they would be recovered. Due to transforming the negative variance component estimations to zero, the distribution of the estimations is positively skewed. Therefore, to evaluate the relative bias in estimates, we calculated the median relative deviation of the estimates from the population value instead of mean relative deviation.
The results show for the traditional approach a high overestimation of the between-case variance, and to a smaller degree of the between-study variance, especially when the number of measurement occasions per phase is only 10. When the number of measurement occasions increases from 10 to 20, the relative bias of estimates decreases significantly. When the number of measurement occasions is small, combining the standardized effect sizes led to better between-study variance estimations compared with synthesizing the standardized raw data, but for larger measurement occasions, the results are more and less similar.
When the Hedges’ correction procedure is applied, underestimation was observed across all conditions, but better estimations were observed with more studies (K=30). Better between-study variance estimates were obtained when using standardized raw data rather than for standardized effect sizes, whereas better between-case variance estimates were obtained when effect sizes rather than raw data were combined.
In this poster, a detailed overview of the performance of different approaches on the estimations’ quality of variance components will be given.

Authors

• Laleh Jamshidi, KU Leuven

• Wim Van den Noortgate, KU Leuven