Comparing Hierarchical Linear Models and Randomization Tests in the Analysis of Multiple Baseline Data

• In Event: Handling Methodological Issues in the Analysis of Single-Subject Experimental Design Data

Abstract

Multiple baseline across subjects designs are single-case experimental designs (SCEDs) in which a treatment is introduced in temporal sequence to different subjects. These designs are often used in the field of education and special education. The power of multiple baseline across subjects designs comes from the possibility to determine whether a systematic change in the independent variable occurs when, and only when, the intervention is directed at the subject in question (cf. Barlow, Nock, & Hersen, 2009). For this poster, we take an exemplary look at the statistical analysis of multiple baseline design (MBD) data presented in the article of Hancock and Kaiser (2002) on the effects of a trainer-implemented Enhanced Milieu Teaching intervention on the social communication of four children with autism. In this article, the participants were assigned randomly to a specific number of baseline data points (i.e., five, six, or seven baseline observations), before the start of the SCED. As such, the design reported on by Hancock and Kaiser (2002) is a randomized MBD. Randomization can strengthen the internal validity of SCEDs because it yields statistical control over known and unknown confounding variables (Edgington, 1975; Onghena, 1992). As such, it enhances the scientific credibility of SCEDs. Additionally, randomization can enhance the statistical conclusion validity of SCEDs because it leads to a statistical test based on the randomization realized in the SCED: the randomization test (RT) (Kratochwill & Levin, 2010; Onghena, 2005).
A more recently developed statistical procedure for analyzing SCED data is hierarchical linear modeling (HLM; e.g., Van den Noortgate & Onghena, 2003a, 2003b, 2008). HLM was developed to analyze clustered data, as in the study of Hancock and Kaiser (2002). According to the Design and Evidence Standards for SCEDs developed by the What Works Clearinghouse (Kratochwill et al., 2010), the use of regression estimates and of multilevel models is promising for analyzing SCED data.
For this poster, we first analyze the empirical data set of Hancock and Kaiser (2002) on Enhanced Milieu Teaching interventions using HLM and RTs, and compare the results of the two analyses. Second, we take the perspective of the applied educational researcher, and summarize the most important advantages and drawbacks of the two methods. Third, we systematically study the statistical strengths and weaknesses of the two methods in a stochastic simulation study. Monte Carlo simulation methods are used to study the appropriateness and accuracy of HLM and RTs under varying circumstances. The simulation study uses parameter values based on empirical SCED studies published in the field of education and special education (a.o. based on a review of SCED studies by Shadish & Sullivan, 2011). As such, the simulation study is substantially inspired and focusses on realistic conditions.

Authors

• Mieke Heyvaert, Katholieke Universiteit Leuven

• Mariola Moeyaert, University at Albany

• Maaike Ugille, K.U. Leuven

• Wim Van den Noortgate, Katholieke Universiteit Leuven

• Patrick Onghena, KU Leuven