Empirical Investigation of the Impact of Unbalanced Moderators in Single-Case Multilevel Meta-Analysis: A Monte Carlo Simulation Study

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Developments

Abstract

Multilevel meta-analysis is a promising approach that can be applied to explain variability in intervention effectiveness between participants in single-case experimental design (SCED) research. This approach allows the inclusion of participant characteristics (e.g., gender, disability types, and race) as moderators to account for some of the variability in intervention effectiveness. Unbalanced moderators are common when synthesizing SCED studies as each unique study has its own specific participant characteristics. However, little is known about the influence of unbalanced moderators in SCED multilevel meta-analysis. Therefore, the goal of this study is to empirically evaluate the robustness of multilevel meta-analysis to unbalanced moderators.
A large-scale Monte Carlo simulation study will be conducted, in which we will generate SCED meta-analytic data with unbalanced moderator variables (taking on values that vary substantially within studies). To reflect the hierarchical structure of SCED meta-analytic data, a model with three levels will be considered to generate the raw data: repeated observations (level 1) are nested within participants (level 2) and participants are nested with studies (level 3). Residuals at the three levels will be assumed to be normally distributed. We will simulate MBDs (the most common SCED), with the start of the intervention staggered across participants. For each outcome variable, within each participant and within each study, the standardized regression-based effect size will be calculated and an adaptation of Hedges’correction factor for small sample bias will be applied. Number of primary SCED studies per meta-analysis will be manipulated using values of 10, 20 or 30. The studies will include 3, 4 or 7 cases, and for each case we will usually generate 10, 20 or 40 measurement occasions. The within-participant variance for each outcome will be fixed to 1, the between-case and between-study variance will be generated to be 0.50 or 2.0. The true size of SCED intervention effects (standardized regression-based coefficient) will typically be generated using a value of 2. The number of moderators included in the model will be set to 1, 2, 3, 4 or 5. The moderators’ values will be sampled from a standardized multivariate normal distribution with study-specific parameters (as moderators’ values vary within studies).
For each condition (i.e., the combination of condition and parameter values), we will simulate 1,000 datasets. SAS (e.g., proc mixed) will be used. All the datasets will be analyzed using the multilevel meta-analytic approach in combination with RVE. Outcomes we plan to investigate will include parameter bias for fixed and random effects (co)variance estimates, variance and mean squared error of the relevant fixed effect parameters, standard error bias (by comparing the mean estimated standard error with the standard deviation of the parameter estimates for each condition), confidence interval coverage, Type I error rates, and power. We will perform ANOVAs on these outcome variables and using practical significance to substantiate “significant”, we will identify the most influential (main and interaction) effects of the simulation design factors. Based on the results of the simulation study, practical implications for SCED meta-analyses including unbalanced moderators will be provided.

Authors

• Mariola Moeyaert, University at Albany - SUNY

• Panpan Yang

• Marzieh Dehghan-Chaleshtori, University at Albany - SUNY

• Wim Van den Noortgate, KU Leuven