On the Multivariate Distribution of Effect Size Estimates From Single-Case Experimental Designs

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Innovations and Complexities

Abstract

Single-case experimental designs (SCEDs) are a class of study designs that use repeated measurement of outcomes on one or a small number of cases in order to evaluate the effects of an intervention. SCEDs comprise a substantial fraction of the evidence base in certain research areas—particularly on topics such as interventions for low-incidence populations, where it is difficult to obtain samples large enough to use between-subjects research designs. Consequently, there is a need for methods to synthesize findings from SCEDs, such as through the use of meta-analysis methods. A variety of methods have been developed for estimating effect sizes and conducting meta-analysis of data from SCEDs, including methods based on within-case effect size metrics, between-case effect size metrics, or raw data synthesis (Pustejovsky & Ferron, 2017). Approaches based on within-case effect size metrics involves first estimating effect sizes for each case in each study, then synthesizing these estimates using multi-level meta-analysis (Van den Noortgate & Onghena, 2008). This generate estimates of an overall average effect, as well as estimates of within- and between-study heterogeneity.
Many different effect size metrics have been proposed for use with single-case designs (Manolov & Moeyaert, 2017; Parker, Vannest, & Davis, 2014). Metrics that have known sampling variances and are suitable for meta-analysis include the within-case standardized mean difference (SMD; Gingerich, 1984), the log response ratio (LRR; Pustejovsky, 2015), and the non-overlap of all pairs (NAP; Parker & Vannest, 2009). These within-case effect size metrics can be used to make comparisons between pairs of phases within an SCED for a single outcome. However, in practice, many SCEDs include multiple outcomes, multiple phases (e.g., baseline, intervention, and modified intervention), or both multiple outcomes and multiple phases. In such studies, it may be useful to estimate multiple effect sizes for inclusion in a meta-analysis. This presents two challenges: a) how to account for the statistical dependency between effect size estimates that are based on a common sample and b) how to model these effect sizes. Well-developed methods are available for addressing these challenges in meta-analysis of between-groups experimental designs (Jackson, Riley, & White, 2011; Moeyaert et al., 2017; Wei & Higgins, 2013), but corresponding methods for SCEDs are lacking.
Draw on the between-groups methodological literature, this study first describes methods for estimating multiple effect sizes, along with corresponding sampling variances and covariances, from SCEDs with multiple outcomes and/or multiple intervention phases. Methods are reviewed for the within-case SMD, LRR, and NAP indices (Lajeunesse, 2011; Wei & Higgins, 2013). A multi-variate, multi-level meta-analysis model is then described and applied to data from an in-progress research synthesis of SCEDs. The example illustrates how the multi-variate meta-analysis approach provides a basis for statistically comparing effects across multiple outcome domains or multiple phases and studying covariance of effects from multiple domains.

Author

• James Eric Pustejovsky, University of Wisconsin - Madison