Meta-Analysis of Single-Case Design Using Multilevel Modeling: Modeling Between-Outcomes Variation in Autocorrelation

• In Event: 51.029 - Handling Methodological Complexities in Effect Size Calculation and Meta-Analysis of Single-Case Experimental Design Data

Abstract

Multilevel modeling (MLM) has recently been suggested for combining single-case experimental design (SCED) data, with promising results (Owens & Ferron, 2012; Ugille, Moeyaert, Beretvas, Ferron, & Van den Noortgate, 2012; Petit-Bois et al., 2016). For example, the fixed effect estimated in a three-level model tends to be reliable and reasonably unbiased when using small sample size. The MLM approach has a great flexibility in that can handle various methodological issues that may arise with single-case studies, such as the need to model possible dependency in the errors, linear or nonlinear trends, and count outcomes (Van den Noortgate & Onghena, 2003).
In meta-analysis of SCED studies, it is common to include similar studies that measure a same construct. However, outcomes of each study may not be exactly same across studies which can lead to include multiple outcomes in a meta-analysis. This could possibly lead dependency of errors, autocorrelation. Generally, it has been found that the estimates of the fixed effects were unbiased but the estimates of variance parameters were substantially biased when autocorrelation is not taken into account (Ferron, Bell, Hess, Rendina-Gobioff, & Hibbard, 2009; Ferron, Dailey, & Yi, 2002; Kwok, West, & Green, 2007; Sivo, Fan, & Witta, 2005; Sivo & Willson, 2000).
Although, the effect of misspecified leve-1 error structure has been well studied in MLM framework, issues of including multiple outcomes in a meta-analysis SCED using multilevel modeling has not received much attention. It is possible that the level-1 error structure may not be same across multiple outcomes. Previous SCED applied meta-analysis studies (Morga, Sideridis, & Hua, 2012; Vanderkerken, Heyvaert, Maes, & Onghena, 2013; Denis, J., Van den Noorgate, Baes, 2011) as well as methodological studies using multilevel modeling (Owens & Ferron, 2012; Ugille, Moeyaert, Beretvas, Ferron, & Van den Noortgate, 2012; Petit-Bois et al., 2016) have assumed the covariance structure is the same for all outcomes. The purpose of this study is to extend the meta-analysis of SCES using multilevel modeling to allow the autocorrelation to vary across multiple outcomes.
In the paper, the model for varying autocorrelations is presented. This model is then applied in the reanalysis of published meta-analysis SCED studies that include more than one outcomes. Results from the varying autocorrelation model are provided along with the results from a model where the dependency in the error structure is assumed constant across outcomes. Results are discussed in terms of the evidence for between-outcome variation in the autocorrelation and in terms of the sensitivity of the treatment effect estimates and inferences to the alternative specifications of the level-1 error structure.

Authors

• Eunkyeng Baek, Texas A&M University - College Station

• Wen Luo, Texas A&M University - College Station

• Maria Antoun Henri, Texas A&M University - College Station