Single-Case Multilevel Meta-Analysis: Handling Multiple Regression Coefficients

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Innovations and Complexities

Abstract

This poster reports on a simulation study that was designed to compare the performance of different approaches to handle dependent effect sizes in single-case experimental design (SCED) research. We focused on the situation where there are two effect sizes per case, more specifically the immediate treatment effect and the effect on a time trend. SCED raw data were simulated with the covariance between the immediate treatment effect and the treatment effect on the time trend at the case and study level. To accommodate these dependent effect sizes, we applied three different approaches: a traditional univariate random effects approach (for each type of coefficients separately) avoiding the existing dependence, a multivariate random effects meta-analytic approach ignoring the existing covariance at case and study levels, and a multivariate multilevel meta-analysis modeling the covariance at higher levels. In the last two approaches, we also studied the performance of the sandwich estimator.
The results show no substantial differences between the underlying models in terms of the bias and MSE of the treatment effect estimates. In terms of the standard errors of the fixed effects, the underlying models along with the number of studies and the interaction of number of measurement occasions, number of cases, and number of studies played a significant role in the variability of the estimates. The results indicate greater underestimation when the sandwich estimator was applied to results from both approaches of modeling and when ignoring the covariances at higher levels. This in turn led to the smaller coverage proportions for results from the models paired with the sandwich estimator, which was slightly smaller than the nominal coverage proportion values.
The between-study and between-case variances are not estimated precisely and accurately in certain conditions. We found underestimated between-study variances across all combinations of conditions. Biased between-study variance estimates were observed particularly in scenarios with 10 or 20 studies and large between-case variances. On the other hand, more bias was observed in the between-case variance estimates. Conditions with a larger between-case variance and 20 or 40 measurement occasions resulted in unbiased between-case variance estimates. Biased estimates were obtained in other conditions. The reported results indicate that the multivariate approach that models the existing covariances does not outperform the multivariate approach that ignores these covariances.
In general, the findings in the current study indicate no substantial differences between the results of applying different three-level meta-analytic models, confirming the robustness of the three-level approach in handling the dependence among the treatment effects at higher levels even if the covariance structure is misspecified. Although biased variance components estimates were obtained in some conditions, the misspecification of the covariance structure did not lead to worse estimates.
In this poster, a detailed overview of the performance of different approaches on the estimations’ quality of overall treatment effects and variance components will be provided.

Authors

• Laleh Jamshidi, University of Regina

• Wim Van den Noortgate, KU Leuven