Credible Data-Generating Models for Single-Case Designs

• In Event: 51.029 - Handling Methodological Complexities in Effect Size Calculation and Meta-Analysis of Single-Case Experimental Design Data

Abstract

A number of parametric models for the analysis and meta-analysis of SCDs have been proposed over the last three and half decades (e.g. Gorsuch, 1983; Crosbie, 1933; Van den Noortgate and Onghena, 2003; J. M. Ferron, Moeyaert, Van den Noortgate, and Beretvas, 2014). One common feature among many of the models propose for SCDs is that they assume that the outcomes are normally distributed, an assumption common to many models used in social science research. In turn, the simulation studies used to examine the properties of these parametric models also generated data that assumed the outcomes in SCDs are normally distributed. This assumption is useful in simulations, because it means that data with arbitrary means and variances can be easily generated. However, it is not entirely clear that this assumption is justified.
A systematic review of SCDs by Shadish and Sullivan (2011) found that over 90% of the outcome variables in those studies were some form of count. Models for counts, such as the Poisson distribution, generally assume a strong relationship between the mean level of a count and its variance. This mismatch between the data found in real SCD studies and the data-generating models used in studies simulating SCD data suggests that the assumptions of many simulations may not be credible, and as a consequence it is difficult to know if the results of these simulations are applicable to real-world data.
The purpose of this study is to examine the credibility of a number of proposed data-generating models for SCDs used in Monte Carlo simulation studies. In the context of a simulation, a credible model is one that is simple enough to understand and manipulate, but also captures all of the important numerical details of the real process it is trying to emulate (Rubinstein & Kroese, 2016). We will examine four data generating models: normally-distributed outcomes with and without autocorrelation, independently distributed Poisson outcomes, quasi-Poisson distributed outcomes using the gamma point-process (Rogosa and Ghandour, 1991), and autocorrelated Poisson outcomes with an AR(1) structure using binomial thinning (McKenzie, 1988).
We will generate data of a fixed phase length as well as variable-length data that is response-guided in nature (Ferron, Joo, and Levin 2017; Ayres and Ledford, 2014), for both multiple-baseline designs and treatment-reversal designs. Response-guided designs involve utilizing outcome data patterns to determine the onset of treatment. We will compare the data generated using these models to the data found in a database complied from seven synthetic reviews SCD data on characteristics such as the mean and variance of the outcomes, as well as the distribution of the phase lengths for variable-length data. In general, we will look to determine which combinations of generating conditions and generating models produce data that is extremely unlike data actually seen in SCD studies. Ruling out particular combinations is an important first step in establishing credible data generating models, and ensuring that the results of existing and future simulation studies are applicable to real data.

Authors

• Daniel Swan, University of Oregon

• James Eric Pustejovsky, The University of Texas at Austin

• Tasha Beretvas, The University of Texas at Austin