Examination of Evidence for Response-Guided Methods in Data From Empirical Examples

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Developments

Abstract

Textbooks on single-case research often articulate response-guided techniques as a standard approach to selecting the timing of interventions. Similarly, Ferron and Jones (2006), the authors noted potentially as many as 80% were response-guided. Potentially is the operative word – many studies lack the explicit documentation that allow researchers to understand the criteria used by the authors to arrive at their decisions about when to intervene. Swan, Pustejovsky and Beretvas (2020) indicated that the within-phase sample variance may be underestimated with respect to the true data-generating process when an SCD is response-guided, and Swan, Pustejovsky and Beretvas (2021) suggested that standard errors from effect sizes will frequently be underestimated when calculated from response-guided data.
Given that response-guided designs may be common, but it is frequently difficult to tell whether or not a particular SCD was response guided, there may be biases in analysis of SCD data that are hard to detect. One method which may provide us more information about the properties of single-case design is to apply the algorithms used in previous studies to empirical data. One of the hallmarks of SCD studies is the availability of the data for analysis and re-analysis, so it is possible to apply response-guided algorithms after the fact to understand whether a study might have used response-guided methods. It will not be possible to definitively say whether or not a study used a particular set of response guided criteria, but it may be possible to identify criteria that were not used or infrequently used. This way it may be possible to narrow down the likely set of criteria used in response-guided designs, and thereby better-understand the potential for bias in the re-analysis of SCD data. Additionally, examining commonalities across studies which may have all used a particular algorithm may help illuminate contextual factors that drove decisions made by researchers about when to intervene. Finally, examining agreement across algorithms might clarify the degree to which apparently different algorithms are actually examining similar data features in practice.
This study will examine the baselines of data drawn from empirical studies using the response-guided algorithms from Swan, Pustejovsky, and Beretvas (2020) to examine three research questions: How frequently are empirical baselines considered stable by each response-guided algorithm? Are outcome or intervention characteristics related to the frequency that empirical baselines are considered stable by each response-guided algorithm? To what extent do the stability judgments made by the response-guided algorithms agree with each other?

Authors

• Daniel Swan, University of Oregon

• James Eric Pustejovsky, University of Wisconsin - Madison

• Tasha Beretvas, The University of Texas at Austin