Effects of Error Structure Specification on the Meta-Analysis of Single-Case Studies of Reading Fluency Interventions

• In Event: Methodological Dilemmas Encountered in Analyses and Meta-Analyses of Single-Case Design Research

Abstract

Multilevel models have been suggested as a method for combining single-case data within and across studies (Nugent, 1996; Shadish & Rindskopf, 2007; Van den Noortgate & Onghena, 2003a,2003b). Van den Noortgate and Onghena (2003) illustrated the use of a two level model to analyze data in the individual studies in single case designs. When including data from multiple studies these models can be extended to three levels: scores may vary over occasions (level 1); across participants from the same study (level 2); and across studies (level 3) (Van den Noortgate & Onghena, 2008).
The multilevel approach allows a large degree of flexibility in modeling the data (Goldstein & Yang, 2000; Hox & de Leeuw, 1997). By using this framework, researchers can make various methodological decisions when specifying a model to approximate the data. Those decisions are particularly important since parameters can be biased if the statistical model is not specified correctly. One of the decisions that must be made involves the choice of variance structure for the errors. In single-case data the potential exists for the non-modeled effects, such as the effect of being ill, to influence multiple consecutive observations in the series leading to autocorrelation. Multilevel modeling allows for this possible dependency of errors which resolves the concern with assuming the errors in the statistical model are independent. However, a critical assumption still exists for multilevel modeling in that the covariance structure may or may not be assumed to be the same for all participants within a study and between studies (e.g., Van den Noortgate & Onghena, 2003a; Ferron, Bell, Hess, Rendina-Gobioff, & Hibbard, 2009; Ferron, Farmer, & Owens, 2010). The purpose of this study is to identify the consequences of error structure specification on the results of a meta-analysis of single-case data involving reading fluency.
In the paper, four distinct models involving different specifications of the error structure were analyzed to understand the impact of this methodological decision on the parameter estimates. The first model was the most traditional model where the dependency in the error structure is assumed constant both within a study and across studies. In the second model the autocorrelation was assumed to be the same across participants within a study, but allowed to vary across studies. In the third model the autocorrelation was assumed to be the same across studies, but allowed to vary across participants within a study. The last model allowed for varying autocorrelation both within and across studies. These models were applied in a meta-analysis of single-case studies involving a common metric of words read correct per minute. Results from the three varying autocorrelation models will be provided along with the results from a traditional model. Results are discussed in terms of the evidence for between-case and -study variation in the autocorrelation, and in terms of the sensitivity of the treatment effect estimates and inferences to the specification of the error structure in a meta-analysis of single-case data.

Authors

• Eun Kyeng Baek, University of South Florida

• Merlande Petit-Bois, University of South Florida

• John M. Ferron, University of South Florida