Recognizing Model Uncertainty Using Bayesian Model Averaging of Effect Sizes for Multiple-Baseline Design Data

• In Event: 51.029 - Handling Methodological Complexities in Effect Size Calculation and Meta-Analysis of Single-Case Experimental Design Data

Abstract

Methodologists have recently derived several parametric effect sizes to quantify an intervention’s effect for single-case experimental design (SCED) studies’ results (for example, Ferron, Moeyaert, Van den Noortgate, Beretvas, 2014; Hembry, Bunuan, Beretvas, & Ferron, Van Den Noortgate, 2015; Maggin, Swaminathan, Rogers, O'Keeffe, Sugai, & Horner, 2011; Hedges, Pustejovsky, & Shadish, 2012; Pustejovsky, 2015; Pustejovsky, Hedges & Shadish, 2014; Shadish, Hedges, Pustejovsky, Boyajian, Sullivan, Andrade, & Barrientos, 2014; Swaminathan, Horner, & Rogers, 2014). For each of these methods, researchers assume a single model for the SCED data’s trends in treatment and baseline phases. This single model translates into an assumption that the probability is one that the single model generated the data. Regardless of whether research judgment or some statistic or fit index is used to select the best model, there is still uncertainty not only in parameter estimates but in the model that is being assumed. Bayesian model averaging (BMA) allows modeling of model uncertainty. Prior probability distributions are not only specified for each parameter but also for the models. BMA could be used to estimate treatment effects for SCED data under each of several model assumptions with each model’s effect size’s posterior distribution then averaged together after weighting them by their model’s posterior probability (e.g., Fragoso & Neto, 2015; Leamer, 1978; Roberts, 1965). The resulting effect size posterior distribution then recognizes model uncertainty and can be used to make inferences about an intervention’s effect without committing to a single, sole model.
Researchers have used the Bayesian framework to estimate treatment effects for SCED studies’ results (e.g., Hembry et al., 2015; Swaminathan, et al., 2014). Moeyaert, Rindskopf, Onghena, and Van den Noortgate (2017) found that with at least five cases per study, Bayesian estimation performed well for recovering parameters in a two-level model for SCED data with measurement occasions (level-1) clustered within each of multiple participants (level-2). However, these studies calculated treatment effects under the assumption of a single (study-specific) model. No research has yet assessed use of BMA for calculating SCED effect sizes that incorporates realistic model uncertainty.
In this study, we demonstrate use of BMA for calculating immediate and delayed effect sizes using participant data from Hetzroni and Tannous’ (2004) multiple baseline design (MBD) study evaluating a computer-based intervention designed to enhance communication skills of five children with autism. We are also conducting a simulation study to evaluate use of BMA under authentic MBD conditions (Moeyaert et al., 2017). The models that were averaged for each simulated participant differed as a function of the trend for intervention phase data (no-, linear, quadratic, and cubic trends). We also use BMA to calculate two kinds of effect sizes including the immediate and delayed (after three time points in the intervention phase) effects. We compare several priors for the set of models (distinguished by the time trend in intervention phase). We assess parameter and standard error bias and the mean squared error of the two kinds of effect sizes and analyses will be completed by the end of August, 2018.

Authors

• Bethany Hamilton, The University of Texas at Austin

• Tasha Beretvas, The University of Texas at Austin

• Mariola Moeyaert, University at Albany