Analyzing Multiple-Baseline Data With Heterogeneity in the Variance Structures: A Monte Carlo Simulation Study

• In Event: Methodological Dilemmas Encountered in Analyses and Meta-Analyses of Single-Case Design Research

Abstract

Recently the use of multilevel modeling to analyze single-case data has gained attention (Nagler, Rindskopf, & Shadish, 2008; Van den Noortgate & Onghena, 2003a, 2003b). Studies have investigated the quality of the fixed effects and variance components, as well as individual effects calculated using empirical Bayes estimates for data with changes in level and/or changes in trends (Ferron, Bell-Ellison, Hibbard, Rendina-Gobioff, & Hess, 2008; Ferron, Farmer, Owens, 2010; Ferron, Owens, & Bell, 2010). More recently, the utility of three-level models have been explored for the meta-analysis of single-case data where individual data are nested within participants who are nested within studies (Owens, 2011). These studies have assumed that the within participant variance is the same across all participnats. However, a review of multiple baseline studies available through the Web of Science database published since 2010 indicates that residual error variance at level 1 is likely to vary across participants.
The purpose of this study was to examine the (1) quality of fixed effect estimates and variance components and (2) the inferences made from these effects when different levels of heterogeneity of the residual error are specified at level 1. Further, it explored the effect of model misspecification on estimates (e.g., model selected assumes no heterogeneity when heterogeneity is present). Monte Carlo simulation methods were used to examine the point and interval estimates of fixed effects and variance components obtained from multiple baseline data. Data were generated based on a two-level model with individual time points nested within participants. At the first level, an outcome (y) was modeled for a participant as a linear function of a single predictor, phase, where phase was a dichotomous variable indicating whether the observation was from the baseline or treatment phase. At level-2 the model allows for variation in the level-1 coefficients across participants. Simulated data conditions were selected based on previous simulation research, and included: (1) number of participants (4, 6, or 8), (2) number of observations in the time series (series length) for each participant (10, 20, or 30), (3) within person variance (no heterogeneity, low heterogeneity, or moderate heterogeneity), (4) autocorrelation (0, .2, or .4), (5) average baseline variance (0.10 or 0.50), and (6) average treatment variance (0.10 or 0.50). For each of these 324 conditions, 5,000 data sets were simulated using SAS IML (SAS Institute Inc., 2005). After each data set was generated, it was analyzed through multilevel modeling using REML estimation via the MIXED procedure in SAS (SAS Institute Inc., 2005). Each data set was analyzed using a multilevel model which assumed no heterogeneity across participants in the residual error variances and one that assumed heterogeneity. Comparisons between the results from the two models were made to determine the effect of model misspecification. Empirical Bayes estimates were obtained by adding the average treatment effect with the individual error for the treatment effect per individual. For the multilevel analyses, interval estimates of the fixed effects and the individual effects were obtained using the Kenward-Roger method for estimating the degrees of freedom.

Authors

• Jennie L. Farmer, Clemson University

• Corina M. Owens, Professional Testing, Inc.