Power to Estimate Moderator Effects in Two-Level Hierarchical Linear Modeling of Single-Case Data

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Innovations and Complexities

Abstract

Previous methodological work indicates that two-level hierarchical linear modeling (HLM) is an appealing framework for the quantitative synthesis of single-case intervention effect sizes across cases. One extension that has not been investigated is the inclusion of moderators in two-level HLM. This study deals with this gap in the literature and empirically validates the appropriateness of HLM to estimate intervention and moderator effects with sufficient power. The significance of the inclusion of moderators lies in their capability to explain variability in estimated intervention effects between cases within a given study. Based on this, inferences can be made about when, why, for whom and under which conditions the intervention is working.
Method. A Monte Carlo simulation study was performed to evaluate the performance of the two-level HLM model with moderators under commonly encountered design conditions. A total of 1,000 replications per condition were generated, resulting in 256,000 datasets. The basic two-level model was gradually extended, going from zero moderator up to three moderators at the second level. The PROC MIXED procedure within SAS (SAS Institute Inc., 2014) was used to analyze the SCED data. ANOVA was used to identify conditions that have a significant impact on the power.
Preliminary Results. The results for the most complex model, including three moderators, are discussed here. The full results will be presented during the conference. When there are only four cases, the model fails to convergence and therefore these results are excluded from analyses. The ANOVA shows that the number of cases has a statistically significant effect on the power to detect both treatment effect and moderator effects. The magnitude of treatment effect significantly influences the power to detect the treatment effect, and the magnitude of each moderator effect significantly impacts the power to detect the corresponding moderator effect. In addition, the interaction between the number of cases and the magnitude of treatment and moderator effects significantly influences the power. Specifically, when the magnitude of treatment effect is small, larger number of cases is needed to have sufficient power. The same interaction pattern is found for the power to detect moderator effects.
Discussion. The preliminary findings show that the magnitude of treatment and moderator effects and the number of cases influence the power. When using a two-level model with three moderators, at least seven cases are needed to identify the true effect. Increasing the number of observations does not increase the power. Furthermore, when all three moderators effects are set to higher values, age and gender require 12 cases to obtain sufficient power, whereas disability requires at least 20 cases. The reason is that the true magnitude of this moderator is small. Detailed results and implications will be discussed at the conference.

Authors

• Panpan Yang, University at Albany - SUNY

• Xinyun Xu, University at Albany-State University of New York

• Esther Kyu Yon Kim, University at Albany - SUNY

• Mariola Moeyaert, University at Albany