A Demonstration and Evaluation Using Single-Subject Experimental Design Studies Data

• In Event: 30.075 - The Meta-Analysis of Single-Subject Experimental Design Data: Methodological Challenges and Approaches

Abstract

Single-subject experimental design (SSED) research is frequently summarized using primarily visual rather than quantitative analyses. Recently, however, there has been a push to synthesize SSED studies using quantitative meta-analytic techniques. While a number of models and methods have been suggested, this study focuses on use of the multilevel modeling framework in which repeated measures (level-1) of participant outcomes are modeled as nested within participants (level-2), and participants within studies (level-3) (see Van den Noortgate & Onghena, 2008; Ugille et al., 2012, Moeyaert et al., 2013).
Use of the multilevel modeling framework for meta-analysis of SSED study results is relatively new (see Van den Noortgate, 2003a; 2003b) and still faces a few methodological dilemmas that require resolution. Of relevance in the current study, some researchers estimate unique variances for level-1 residuals in the baseline and intervention phases, others assume homoscedasticity. The current study focuses on deriving a method to inform decisions about the homoscedasticity of level-1 residuals. This work adapts Hedges and Hedberg’s idea for their Variance Almanac (2011) for group comparison study research to provide methods that can be used to build a similar almanac to inform SSED research.
To handle the different scales for outcomes in different studies it would be useful to have reasonable synthesized values of variance ratios rather than of variances. The ability to synthesize variance ratios can inform understanding of patterns of heteroscedasticity in the baseline and intervention phase level-1 residuals. However, no work has been found that explores meta-analysis of variance ratios.
To use inverse-variance weights for the meta-analysis, we first derived the sampling error variance for a variance ratio using Taylor series approximations and properties of the multivariate Wishart distribution (Muirhead, 1982). The final paper will demonstrate synthesis and interpretation of intervention to baseline level-1 residuals’ variance ratios using a real SSED meta-analysis dataset (Carr, Moore, & Anderson, 2014) using a two-step process in which 1) the variance ratio for the series for each participant per study is estimated, and then 2) the variance ratios are synthesized using a three-level random effects meta-analysis model with variance ratios (level-1) clustered within participants (level-2) within studies (level-3). We also demonstrate how to assess moderators that might explain variability in variance ratios.
In addition to the real data analysis, we have begun a simulation study that assesses recovery of the true variance ratio under a number of manipulated conditions. (The simulation conditions will be completed by the end of September with results summarized by the end of November). The conditions that we are manipulating include: the number of studies, participants per study, and measurement occasions per phase and series, the true variance ratio, and the degree of variability in the variance ratios across studies and participants. Data are being generated to fit a three-level change-in-levels model (Ugille et al., 2012). Variance ratios are being estimated for each series and then synthesized across participants and studies. Relative parameter and standard error bias are being captured and guidelines for optimal recovery of variance ratios will be provided.

Authors

• Daniel Peche Gonzalez, The University of Texas - Austin

• Susan Natasha Beretvas, The University of Texas - Austin

• Christopher Runyon, The University of Texas - Austin