The Consequences of the Misspecification of the Level 1 Error Structure When Meta-Analyzing Single-Case Designs Using a Three-Level Model

• In Event: 61.033 - Single-Case Experimental Designs: Developments in Statistical Analysis, Effect Size Metrics, and Meta-Analysis Methods

Abstract

Single-case interventions allow for the repeated measurement of a case or participant across multiple time points, to assess the treatment’s effect on one specific case or participant. The basic interrupted time series design includes two phases: baseline and treatment. Raw data from a set of single case design studies have a similar structure. In 2008, Van den Noortgate and Onghena proposed that if raw data from several single case designs are used in a meta-analysis, scores can be varied at each of the three levels: over occasions (level-one), across participants from the same study (level-two), and across studies (level-three).

The multi-level approach allows for a large degree of flexibility in modeling the data (Goldstein & Yang, 2000; Hox & de Leeuw, 1997). Researchers can make various methodological decisions when specifying the model to approximate the data. Those decisions are critical since parameters can be biased if the statistical model is not correctly specified. The first of these decisions is how to model the level-one error structure--is it correlated or uncorrelated? Recently investigations of the Van den Noortgate and Onghena’s (2008) three-level meta-analytic model have shown that the fixed effects tend to be unbiased but the variance components have been problematic across a range of conditions (Owens & Ferron, 2011; Ugille, Moeyaert, Beretvas, Ferron, & Van den Noortgate, 2012). However, based on a thorough literature review, no one to date has examined the use of fit indices or log likelihood tests to select an appropriate level-one error structure for the three-level meta-analytic model. In other areas of application, investigations of the use of fit indices to select the level-one error structure have produced varying results (Ferron, Dailey, & Yi, 2002; Gomez, 2005; Kesselman et al., 1999; Singer & Willet, 2003).

The purpose of the study was two-fold: 1) to determine the extent to which the various fit indices can correctly identify the level-one covariance structure; and 2) to investigate the effect of various forms of misspecification of the level-one error structure when using a three-level meta-analytic single-case model. This study used Monte Carlo simulation methods to address the aforementioned research questions. Multiple design, data, and analysis factors were manipulated in this study. The study used a factorial design with seven experimental variables: 1) The number of primary studies per meta-analysis; 2) The number of participants per primary study; 3) The series length per participant; 4)Variances of the error terms; 5) The levels for the treatment effects; 6) Various types of covariance structures were used for data generation; and 7) The form of model specification.

The results of this study indicate that the fixed effects tend to mostly be unbiased; however, the variance components were extremely biased with particular design factors. The study also concluded that the use of fit indices to select the correct level-1 structure was appropriate for certain error structures. There were multiple implications for the applied single-case researcher, applied meta-analyst and for the methodologist.

Author

• Merlande Petit-Bois, Pinellas County Schools