Impact of Imbalances in Single-Case Multilevel Analysis: A Monte Carlo Simulation Study (Poster 5)

• In Event: Methodological Innovations and Complexities in the Analysis and Meta-Analysis of Single-Subject Experimental Design Data

Abstract

Multilevel meta-analysis is a promising approach that can be applied to explain variability in intervention effectiveness between participants and/or studies in single-case experimental design (SCED) research. This approach allows the inclusion of participant (e.g., gender, disability types, and race) and/or study characteristics (e.g., quality and setting) as moderators to account for some of the variability in intervention effectiveness. Little is known about the performance of two-stage HLM for meta-analysis with the inclusion of unbalanced moderators. Therefore, the goal of this study is to empirically evaluate the statistical properties of this model through a large-scale Monte Carlo simulation study.
To reflect the hierarchical structure of SCED meta-analytic data, a three-level model will be used to generate the raw data: repeated observations (level 1) are nested within participants (level 2) and participants are nested within studies (level 3). A varying number of moderators and degrees of imbalance at the participant and study level will be included. Residuals at the three levels will be assumed to be normally distributed. Number of primary SCED studies per meta-analysis will be manipulated using values of 10, 20 or 30. The studies will include 3, 4 or 7 participants, and for each participant we will generate 10, 20 or 40 measurement occasions. The within-participant variance for each outcome will be fixed to 1, and the between-participant and between-study variance will be generated to be 0.50 or 2.0. The true size of SCED intervention effect (standardized regression-based coefficient) will be generated using a value of 2. The number of moderators included in the model will be set to 1, 2 or 3 at level 2 and level 3. The values of the moderator effects and the degree of imbalance of the moderators will take on similar values as in the simulation study by Moeyaert et al. (2021). For each condition (i.e., the combination of condition and parameter values), we will simulate 1,000 datasets. All the simulated datasets will be analyzed using the two-stage HLM approach (combining standardized regression-based effect size with Hedges’ correction factor).
We will investigate parameter bias, mean squared error, standard error bias, confidence interval coverage, Type I error rates, and power of the intervention and moderator effects. We will use generalized linear modeling on these statistical properties to identify the most influential main and interaction effects of the simulation design factors (partial eta-square statistics will be used to substantiate “significant” instead of p-values). Based on the results of the simulation study, recommendations and implications for the field of SCEDs will be discussed.

Authors

• Yaosheng Lou, University at Albany - SUNY

• Mariola Moeyaert, University at Albany - SUNY

• Panpan Yang, Princeton University

• Yukang Xue, University at Albany - SUNY