The Impact of Response-Guided Designs on Within-Case Effect Size Estimates From Count-Outcome Treatment Reversal Designs

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Innovations and Complexities

Abstract

Single-case researchers commonly conduct studies using response-guided designs, in which the length of each phase is not a fixed element of the design. Rather, in a response-guided design, choices about when to transition from the current phase to the next phase are based, at least in part, on features of the data series, such as the mean level, trend, and variability observed within the current phase. The approach helps to ensure that the resulting data will allow for a straightforward visual assessment of whether a functional relation exists between the intervention and the outcome. Ferron and Jones (2006) found that of 101 SCDs they reviewed, potentially as many as 80% were response-guided, and textbooks on single-case research describe response-guided techniques as the standard approach.
Given that response-guided designs are common, it is important to understand how their use influences the data produced by SCD studies. Recent work by Swan et al. (2020) indicated that the within-phase sample variance may be underestimated with respect to the true data-generating process when an SCD is response-guided. This has potential implications for any effect size estimate which depends explicitly on the sample variance for scaling, such as the within-case standardized mean difference (SMD); for effect sizes based on the degree of overlapping data across phases, such as the non-overlap of all pairs (NAP); and for the standard errors of the effect sizes estimates from response-guided data, even when the effect size does not depend upon the sample variance for scaling, such as the log-response ratio (LRR).
In this study, we use Monte Carlo simulation to examine the impact of response-guided designs on simple within-case effect sizes. Considering that many SCD outcomes are some form of a count (Shadish & Sullivan, 2011), along with the centrality of sample variability to judgements in response-guided designs, the simulations focus on Poisson-distributed outcomes, a data-generating model for counts where the mean and variance are closely linked. We simulate ABAB designs with no within-phase trend, with phase lengths that have been determined by response-guided algorithms drawn from Swan et al. (2020). For comparison, we also simulate designs with fixed phase lengths, which are not response guided. For each simulated data series, we estimate three effect sizes: the within-case SMD, the NAP, and the LRR. We believe that these simple effect sizes are likely to be used in practice by applied researchers, and so our results may have immediate relevance for their work. We examine the biases of the effect sizes estimates and their standard errors within and across different simulated response-guided methods and the fixed-length designs in order to better-understand the potential impact of response-guided experimentation on treatment effects estimated from those designs.
We find that the response-guided methods can cause notable biases in treatment effect estimates of the within-case SMD, while the LRR and NAP are largely unaffected by response-guided methods. We also find that response guided methods can lead to underestimates of the standard errors for all three effect sizes.

Authors

• Daniel Swan, University of Oregon

• James Eric Pustejovsky, University of Wisconsin - Madison

• Tasha Beretvas, The University of Texas at Austin