Meta-Analysis of Single-Case Experimental Designs Using Robust Variance Estimation

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Innovations and Complexities

Abstract

Single-case experimental designs (SCEDs) involve measuring an outcome repeatedly over time across two or more distinct phases, such as a baseline phase and an intervention phase. With such data, the effect of intervention is assessed by comparing the data series during the baseline phase with the series during the treatment phase(s) within each case, so that each participant serves as their own control (Horner & Odom, 2014). In bodies of literature where there are many SCEDs examining a common class of intervention, meta-analysis methods may be used to synthesize results across studies and draw conclusions about the overall effects of intervention, about variability in effects, and about factors associated with larger or smaller effects (Pustejovsky and Ferron, 2017). Because of the structure of SCEDs, in which multiple cases are nested within studies, syntheses of SCED should make use of multilevel meta-analytic models (Van den Noortgate & Onghena, 2008).
One approach to quantifying the direction and magnitude of intervention effect for SCEDs is through within-case effect size measures. Among the within-case effect size measures, the non-overlap of all pairs (NAP; Parker & Vannest, 2009), the within-case standardized mean difference (WC-SMD; Busk & Serlin, 2015; Gingerich, 1984), and the log response ratio (LRR; Pustejovsky, 2018) are suitable for meta-analysis because they have known sampling variances, allowing researchers to make statistical inference about the intervention effects. Available formulas for the sampling variances of NAP, WC-SMD, and LRR are all based on the assumption that outcome measurements are independent within each measurement phase. This assumption would be violated if outcomes are instead auto-correlated—a plausible concern given that the outcomes represent repeated measures on a single unit. In the presence of positive autocorrelation, the sampling variances will be estimated downward, which could lead to inflated type I error rate and inaccurate statistical inferences (Beretvas & Chung, 2008; Petit-Bois, et al., 2016). However, the implications of auto-correlation for meta-analysis of these effect sizes has yet to be examined.
Robust variance estimation (RVE, Hedges, Tipton, & Johnson, 2010; Tipton, 2015; Tipton & Pustejovsky, 2015) has been widely used in the meta-analysis of group experimental designs to estimate unbiased standard errors for overall average treatment effect estimates and meta-regression coefficients. RVE has been proposed as a solution for meta-analysis of SCEDs when sampling variances of effect sizes might be estimated inaccurately due to autocorrelation (Pustejovsky, 2018), but its performance has yet to be assessed. In this study, we use simulation to evaluate the performance of multi-level meta-analysis, with and without use of RVE, in the presence of autocorrelation when using NAP, WC-SMD, and LRR for meta-analyzing SCEDs. We examine the bias of overall average effect size estimates, standard errors, and variance component estimates, as well as the calibration of confidence intervals for overall average effect sizes.

Authors

• Man Chen, University of Texas at Austin

• James Eric Pustejovsky, University of Wisconsin - Madison