Performance of Multilevel Modeling and Power of Tests for Variance Components in the Analysis of Single-Case Experimental Designs

• In Event: The Analysis and Meta-Analysis of Single-Case Experimental Data

Abstract

There is a growing consensus that the serial dependency among errors of observations within a case over multiple time points should not be ignored in SCEDs. Typically, outcome scores that are measured closer in time are more related to each other than outcome scores measured further away in time, commonly referred to as autocorrelation. Ignoring autocorrelation may result in inflated Type I error rates for tests of treatment effects, diminished confidence interval coverage for the estimates of treatment effects (Greenwood & Matyas, 1990; Toothaker, Banz, Noble, Camp, & Davis, 1983), and biased variance component estimates (Ferron, Bell, Hess, Rendina-Gobioff, & Hibbard, 2009; Ferron, Dailey, & Yi, 2002; Kwok, West, & Green, 2007).
Autocorrelation is difficult to estimate in brief time series. Researchers must account for large sampling error within autocorrelation estimation when working with SCEDs, which typically have a small number of time points within a case (Huitema & McKean, 1991; Ferron, 2002; Shadish, Rindskopf, Hedges, & Sullivan, 2013). Conventional estimation methods based on large sample approximation or strong distributional assumptions, such as restricted maximum likelihood (REML), do not perform adequately in such scenarios. In the present study, we explore bootstrapping as an alternative estimation method for adjusting for autocorrelation in the analysis of SCED through multilevel modeling. The main goal is to improve accuracy and precision of the autocorrelation estimate and variance components.
The bootstrap procedure investigated in this study consists of two stages. In stage 1, an adjusted autocorrelation is estimated for each participant using the double-bootstrap method developed by McKnight, McKean, and Huitema (2000). This method leads to estimates of the degree of bias of autoregressive parameters and thus the adjusted autocorrelation will be used in the follow up model specification. In stage 2, because of the inherent attribute of the small sample size of SCED, multilevel bootstrapping methods are used to obtain bootstrap samples that model the dependence and auto-correlations of observations. Three multilevel bootstrapping methods are considered. The first method was developed by Carpenter, Goldstein, and Rasbash (2003), which reflates and independently resamples level-1 and level-2 residuals. The second method is the linked residual bootstrap (Goldstein, 2011), which draws the level-1 residuals with replacement from the corresponding level-2 unit that has been drawn. The third method is the moving block bootstrap, which preserves the dependence among observations by resampling a block of estimated level-1 residuals (Ju, 2015).
The performance of the two-stage bootstrapping procedure is evaluated using a Monte Carlo study that examines various conditions such as series length, the number of cases, and the degree of autocorrelation. For each condition, parameter bias, RMSE and CI coverage and width are being estimated. Simulations are estimated to be finished in October, 2019.

Authors

• Haoran Li, Texas A&M University - College Station

• Kwok Hap Lam, Texas A&M University - College Station

• Wen Luo, Texas A&M University - College Station

• Eunkyeng Baek, Texas A&M University - College Station

• Christopher Glen Thompson, Texas A&M University - College Station

• Ashley Majzun, Texas A&M University - College Station