Type I Error and Power of Hierarchical Linear Models Versus Additive Combination of Randomization Tests for Intervention Effects in Single-Case Experiments: A Monte Carlo Simulation Study

• In Event: 61.033 - Single-Case Experimental Designs: Developments in Statistical Analysis, Effect Size Metrics, and Meta-Analysis Methods

Abstract

Objective. A considerable number of studies published in the domain of education rely on single-case experimental designs. In this poster, we report on a Monte Carlo simulation study, comparing hierarchical linear modeling (HLM; e.g., Van den Noortgate & Onghena, 2003a, 2003b, 2008) and using the additive method to combine randomization test p values (RTcombiP; Edgington, 1972; Onghena & Edgington, 2005) for single-case experimental data. We evaluated Type I error rate control and statistical power of HLM and RTcombiP as applied to replicated, randomized AB designs.

Methods. We simulated raw data for replicated randomized AB designs. In accordance with the Design and Evidence Standards developed by the What Works Clearinghouse (Kratochwill et al., 2010), we simulated the data in such a way that each phase included at least three data points. The data were sampled from a normal distribution. Two factors were kept constant: the mean baseline effect was 0 and the within-case variance was 1. Four factors were manipulated: (1) the mean intervention effect was 0 or 2; (2) the number of cases included in a study was 3, 4, 5, 6, or 7; (3) the number of measurement times for each case within one study was 10, 20, 30, or 40; and (4) between-case variance was 0, 0.1, 0.3, 0.5, 2, 4, 6, or 8. In the simulation study, we compared: (1) the p value of the estimated intervention effect across cases in a two-level HLM (using restricted maximum likelihood estimation; Littell, Milliken, Stroup,Wolfinger, & Schabenberger, 2006), and (2) the p values of RTs (with absolute difference between the phase means as the test statistic for the RT) combined using the additive method (Edgington, 1972; Onghena & Edgington, 2005). All tests were performed at the 5% significance level.

Results. For HLM, the Type I error rate was smaller than .05 in most conditions, whereas for RTcombiP the Type I error rate was smaller than .05 in all conditions. For the conditions with a small number of observations RTcombiP proved to be a very conservative test. For both HLM and RTcombiP a larger number of combined cases resulted in higher statistical power. Smaller values for the between-case variance resulted in higher statistical power for HLM. A larger number of data points resulted in higher statistical power for RTcombiP.

Conclusions. The Type I error rate results are reassuring for the single-case meta-analyst: Both HLM and RTcombiP provide a valid test of the intervention effect. Our results imply that including four cases in a replicated randomized AB design may already be sufficient for analyzing the data by means of HLM and RTcombiP, when the between-case variance is low (i.e., 0.5 or less) for HLM, and when the number of data points for the included cases is large (i.e., 30 or more) for RTcombiP. The conclusions of this study are limited to the conditions that were simulated: The data were sampled from a normal distribution, there was no trend in the data, and the data were not autocorrelated.

Authors

• Mieke Heyvaert, Katholieke Universiteit Leuven

• Mariola Moeyaert, University at Albany

• Maaike Ugille, K.U. Leuven

• Wim Van den Noortgate, Katholieke Universiteit Leuven

• Patrick Onghena, KU Leuven