Impact of Within-Case Variability on Tau Indices and the Hierarchical Linear Modeling Approach for Multiple-Baseline Design Data: A Monte Carlo Simulation Study

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Innovations and Complexities

Abstract

Modeling single-case design data can be complex due to data characteristics inherent to the design. One such data characteristic is within-case variability, or the amount that individual data points differ from each other within one individual case (i.e., within-case variance). When within-case variance is present in a data set, this can affect outcome estimates. For this paper, a parametric method [i.e., hierarchical linear modeling (HLM) as a regression-based model (e.g., Van den Noortgate & Onghena, 2003a, 2003b, 2007)], and a nonparametric method [i.e., Tau and its three variants (Parker et al., 2011; Tarlow, 2017)] are used to compare the effect of within-case variance on the estimates of multiple-baseline design data.
Research Aims. The research aim is to investigate the degree to which the three Tau variants [i.e., Tau-U, Tau-U Trend A (Parker et al., 2011) and Tau-U adjusted (Tarlow, 2017)] and the HLM approach are robust against different values of the within-case variance by using a Monte Carlo simulation.
Method. A Monte Carlo simulation study was conducted with 216 unique design conditions, and with 1,000 replications per condition. The simulated data was generated using a two-level HLM model. The same two-level HLM model was applied to calculate the intervention effect, and the three Tau indices were also applied to calculate the intervention effect. A two-way factorial Analysis of Variance was used to evaluate and isolate the design factors that have a large impact on the three Tau estimates and the HLM based effect size estimates (with special attention to the influence of the within-case variance). Data was generated and analyzed using RStudio (v0.98.1074; RStudio Team, 2015).
Results. The results of this study demonstrate that the three Tau indices generally perform well when the within-case variance is small relative to the magnitude of the non-zero true intervention effect, but do not perform as well when the within-case variance is large relative to the magnitude of the non-zero true treatment effect. Other factors such as measurement occasions, number of cases, and trend also affect the Tau estimates. HLM does not appear to be significantly affected by within-case variance or the other factors, making it a good alternative and/or complement to the aforementioned Tau variants.
Conclusion. This study enhances the methodology of SCED research by investigating the impact of design conditions typically encountered in the field of SCEDs on measures such as Tau variants and HLM-based effect sizes. A flowchart is created based upon the results to help researchers use Tau indices more appropriately. As SCEDs become increasingly popular, it is likely researchers will continue to quantify effects rather than rely on visual analysis. Therefore, it is crucial that the statistical properties of effect measures are investigated.

Authors

• Joelle Fingerhut, Rutgers University

• Xinyun Xu, University at Albany-State University of New York

• Mariola Moeyaert, University at Albany