Multilevel Meta-Analysis of Single-Case Study Results: A Simulation Study

• In Event: Methodological Dilemmas Encountered in Analyses and Meta-Analyses of Single-Case Design Research

Abstract

In applied single-case experimental research, various replication strategies are used within and over studies to investigate the generalizability of the results (Barlow & Hersen, 1984; Ferron & Scott, 2005). Van den Noortgate and Onghena (2003a, 2008) describe one approach that can be used for a systematic and statistical synthesis of the data from several cases: the use of multilevel models. Because case studies often comprise a few cases, and cases are repeatedly measured, we propose the use of a three-level model, describing the variation at three levels: the scores can vary over measurement occasions within cases (first level), over cases from the same study (second level) and over studies (third level). The three-level model can be regarded as a multilevel extension of regression models that are used for describing or analyzing the data from one case, such as the model of Center, Skiba and Casey (1985-1986). A major strength of the model is its flexibility: adaptations can be made in order to model autocorrelation, characteristics of the design and the moderating effect of study and case characteristics, or to analyze measures of effect size instead of raw data.
Although extensions have been proposed for a variety of situations and the approach has been illustrated using real data, the behavior of the multilevel approach is not yet fully understood. An open question is for instance how appropriate the multilevel approach is for the small sample sizes that are often encountered in this kind of research. Therefore, this poster will report on an extensive simulation study, empirically investigating the performance of the multilevel approach for a variety of sample sizes at each of the levels (i.e., the number of observations per case, the number of cases per study and the number of studies) and of the between-case and between-study variances. The study reveals minimal sample sizes at each of the levels that are required to make valid and reliable inferences, and suggests situations for which improvements of the basic three-level models and/or the maximum likelihood parameter estimation methods are required.

Authors

• Wim Van den Noortgate, Katholieke Universiteit Leuven

• Mariola Moeyaert, Katholieke Universiteit Leuven

• Maaike Ugille, Katholieke Universiteit Leuven

• Patrick Mjl Onghena, Katholieke Universiteit Leuven