Misspecification Issues in Multilevel Modeling of Single-Case Experimental Designs: Empirical Illustrations

• In Event: 61.033 - Single-Case Experimental Designs: Developments in Statistical Analysis, Effect Size Metrics, and Meta-Analysis Methods

Abstract

This study provides empirical illustrations of three possible misspecification issues in the context of multilevel analysis of single-case experimental designs (SCEDs), which have previously investigated using Monte Carlo simulation studies. Previous research indicates that it is not always obvious how to (1) specify the covariance matrix, (2) code complex single-case designs or (3) model dependent effect sizes in alternating treatment designs (ATD). However, this can have far-reaching consequences, as misleading study results are obtained on which important policy, research and practice decisions are based. To illustrate the consequences of ignoring existing covariance and to code complex single-case designs, we used a re-analysis of the meta-analysis of SCED studies of Heyvaert, Saenen, Maes, and Onghena (2014), a meta-analysis that includes different types of SCED studies. The modeling of dependent effect sizes is illustrated by another dataset as we focus on dependency between treatment effects within ATDs. We created this dataset ourselves and collected ATD studies published in 2012 focusing on treatments to increase reading fluency for children with autism.

The results indicate that the ignoring existent covariance has no effect on the estimated average treatment effects in contrast to the covariance estimates. The estimation of the average treatment effect across different types of SCED studies is dependent on the analysis model (i.e., univariate vs multivariate). The multivariate model takes the dependence between effect sizes into account and has multiple advantages over univariate models, including more information and smaller standard errors. Finally, because ATDs are characterized by multiple treatment effects within one subject (Barlow & Hayes, 1979; Barlow, Nock, & Hersen, 2009), these effect sizes are dependent and therefore the sampling error of the multiple effect sizes within a study will be dependent (Hedges, Tipton, & Johnson, 2010). Previous research commonly ignored this issue of dependency (Cheung & Chang, 2004) which results in standard errors that are underestimated and confidence intervals that are too small (and therefore in an increased risk for Type I error). In this part, we compare univariate and multivariate multilevel modeling with the robust variance estimation method. Primary results of the Monte Carlo simulation study are in favor of the robust variance estimation both in terms of the bias, mean squared error, and coverage proportion of the treatment effect and variance component estimates. The empirical illustration is estimated to be finished in October, 2014.

This study illustrates the degree to which study results can be dependent on different modeling options. Only three misspecification issues are discussed here, yet many others are possible, such as misspecification of the trajectory, autocorrelation, external event effects, etc. However, this study already gives some insight in commonly encountered misspecification issues.

Authors

• Mariola Moeyaert, University at Albany

• Maaike Ugille, K.U. Leuven

• Patrick Onghena, KU Leuven

• Mieke Heyvaert, Katholieke Universiteit Leuven

• Wim Van den Noortgate, Katholieke Universiteit Leuven