Estimating Instantaneous and Maximum Treatment Effect Sizes for Nonlinear Treatment-Phase Trajectories in Single-Subject Experimental Design Studies

• In Event: 30.075 - The Meta-Analysis of Single-Subject Experimental Design Data: Methodological Challenges and Approaches

Abstract

Visual analysis has typically been used to make inferences about a treatment’s effectiveness in single-subject experimental design (SSED) studies. However, methodologists are increasingly encouraging use of statistical models to quantify and synthesize treatment effects. We focus on use of the multilevel modeling framework because it offers a flexible framework for meta-analysis of standardized SSED treatment effects (Moeyaert, et al., 2013a & 2013b). Recent research has challenged the assumption of flat intervention phase data and demonstrated use of non-linear models for treatment phase trajectories (Hembry et al., 2015; Shadish et al., 2013). A change-in-levels effect size is inappropriate for capturing a treatment effect given a linear trend in treatment phase data. In this study, we suggest two alternative effect sizes – the immediate effect and the maximum effect - that align with the kinds of effects sought visually by applied SSED researchers and that can be captured regardless of most intervention phase trends. Using real meta-analytic SSED data, we demonstrate how each of these effects can be calculated from data fitting each of the following functional forms for treatment phase data, including: change-in-levels only, linear, quadratic, cubic and logistic trends and a change-point model. (Note, no maximum effect is calculable given a linear trend). We also conduct a simulation study to assess parameter and standard error recovery for the two effects for each model mentioned.
The real data used have been extracted from published meta-analyses of SSED studies that examines self-management interventions for students with autism (Carr, Moore, & Anderson, 2014). For this initial investigation into these effect sizes, we used only the initial baseline phase and the first treatment phase data per series to calculate each effect size. For all data the root mean squared error (RMSE) from the baseline phase data was used to standardize the data for the entire series (Van den Noortgate & Onghena, 2008). We also used the bias correction reported in Ugille et al.’s study (2014).
A fully balanced simulation study has also been started with analyses that will be completed by the end of August and results summarized by the end of October. In the fully balanced simulation study, we manipulated the following conditions: functional form of treatment phase trajectory, magnitude of immediate and maximum effects, number of measurement occasions, participants per study, and studies being synthesized, and mixture of true functional forms for studies’ series within the simulated meta-analysis.
Relative parameter and standard error bias are being captured for estimates of the immediate and maximum effect estimated using each of the models listed earlier. In addition, performance of two methods for identifying each series’ optimal functional form are being compared using the proportion of correct model identifications as well as the resulting recovery of the immediate and maximum effects. The two methods include selection of the functional form for each series based on 1) the functional form with the minimum RMSE and 2) the functional form with the lowest information criteria (AIC, BIC). The analysis and discussion of the results will be presented.

Authors

• Christopher Runyon, The University of Texas - Austin

• Susan Natasha Beretvas, The University of Texas - Austin