Using Multilevel Modeling to Analyze Multiple Baseline Data With Heterogeneity in Variance Structures: A Monte Carlo Simulation Study

• In Event: 61.033 - Single-Case Experimental Designs: Developments in Statistical Analysis, Effect Size Metrics, and Meta-Analysis Methods

Abstract

Recently the use of multilevel modeling to analyze single-case data has gained attention (Nagler, Rindskopf, & Shadish, 2008; Van den Noortgate & Onghena, 2003a, 2003b). Studies have investigated the quality of the fixed effects and variance components, as well as individual effects calculated using empirical Bayes estimates (Ferron, Bell-Ellison, Hibbard, Rendina-Gobioff, & Hess, 2008; Ferron, Farmer, Owens, 2010; Ferron, Owens, & Bell, 2010). More recently, the utility of three-level models have been explored for the meta-analysis of single-case data, where individual data are nested within participants who are nested within studies (Owens, 2011). These studies have assumed that the within participant variance is the same across all participants. However, a review of multiple baseline studies available through the Web of Science database published since 2010 indicates that residual error variance at level 1 and level 2 are likely to vary across participants. Farmer and Owens (2012) determined that, overall, within-person variance did not impact bias of fixed effects and the confidence interval coverage or widths were acceptable. However, the impact of between-person heterogeneity has not been investigated for multiple baseline studies.

The purpose of this study was to examine the (1) quality of fixed effect estimates and (2) the inferences made from these effects when different levels of heterogeneity of the residual error are specified at level 1. Further, it explored the effect of model misspecification on estimates (e.g., model selected assumes no heterogeneity when heterogeneity is present). Monte Carlo simulation methods were used to examine the point and interval estimates of fixed effects obtained from multiple baseline data. Data were generated based on a two-level model with individual time points nested within participants. At the first level, an outcome (y) was modeled for a participant as a linear function of a single predictor, phase, where phase was a dichotomous variable indicating whether the observation was from the baseline or treatment phase. At level-2 the model allows for variation in the level-1 coefficients across participants. Simulated data conditions were selected based on previous simulation research, and included: (1) number of participants, (2) number of observations in the time series (series length) for each participant, (3) within person variance (no heterogeneity, low heterogeneity, or moderate heterogeneity), (4) between person variance (no heterogeneity, low heterogeneity, or moderate heterogeneity), (5) autocorrelation, (6) average baseline variance, and (7) average treatment variance. For each of these 972 conditions, 5,000 data sets were simulated using SAS IML (SAS Institute Inc., 2011). After each data set was generated, it was analyzed through multilevel modeling using REML estimation via the MIXED procedure in SAS (SAS Institute Inc., 2011). Each data set was analyzed using a multilevel model which assumed no heterogeneity across participants in the residual error variances and one that assumed heterogeneity. Comparisons between the results from the two models were made to determine the effect of model misspecification.

Authors

• Jennie L. Farmer, Clemson University

• Corina M. Owens, Professional Testing, Inc.