Multilevel Meta-Analysis of Alternating Treatment Design Studies: A Monte Carlo Simulation Study

• In Event: Handling Methodological Issues in the Analysis of Single-Subject Experimental Design Data

Abstract

Previous simulation studies have focused on empirical validations of the three-level model proposed by Van den Noortgate and Onghena (2008) to synthesize multiple-baseline design (MBD) data over cases and over studies (e.g., Moeyaert, et al., 2013). However, quantitative synthesis of results from alternating treatment designs (ATDs) - which are second in frequency of use to MBDs (Shadish &, Sullivan, 2011) – has not yet been explored. In order to understand characteristics of ATDs, we first reviewed all published ATD studies in 2011 and 2012. (A full description of the review will be provided in the final paper). Of particular importance, we found that a large majority of ATDs involved a baseline and two alternating intervention phases.
For calculating ATD effect sizes we propose using an ordinary least square (OLS) per-case analysis based on the design matrix proposed by Moeyaert, et al. (2013) in which three time variables are defined (one for each of the baseline, treatment 1 and treatment 2 phases). This allow calculation of the case-specific immediate treatment effect for the first and the second treatments. The within-baseline phase residuals’standard deviation is also estimated using the same OLS per-case analysis which is then used to standardize the data.
Special attention should be paid to these two treatments effect sizes because each treatment is being compared with a common baseline and the resulting effects are not independent. Once two effect sizes are obtained, we then propose using the Cholesky transformation to make the effect sizes independent (Kalaian & Raudenbush, 1996). In the next step the effect sizes should be standardized using the estimated within-baseline phase standard deviation from the OLS analysis (Van den Noortgate & Onghena, 2008). Then Hedges (1981) small sample size bias correction should be applied (Ugille et al., 2013). Last, we propose simply combining the Cholesky-transformed, standardized, and bias-corrected effect sizes using the three-level meta-analysis model proposed by Van den Noortgate and Onghena (2008).
We conducted a Monte Carlo simulation study to empirically validate the suggested method for the synthesis of ATD data. The conditions that were manipulated were based on the review of applied ATDs and included the number of level-1, level-2, and level-3 units, the true values of the immediate treatment effects, the within-case variance, the between-case variance and the between-study variance. We generated 2,000 datasets for each combination of conditions. The proposed method was evaluated by calculating the bias, the mean squared error, the standard deviation and the coverage proportion of the 95% confidence intervals of the fixed effects. The final paper will present far more detail about the proposed method for synthesizing ATD results along with its justification as well as providing and discussing the implications of the results.

Authors

• Mariola Moeyaert, University at Albany

• Rommel Lazo Bunuan, The University of Texas - Austin

• Susan Natasha Beretvas, The University of Texas - Austin