An Examination of Measurement Procedures and Baseline Behavioral Outcomes in Single-Case Research

• In Event: 51.029 - Handling Methodological Complexities in Effect Size Calculation and Meta-Analysis of Single-Case Experimental Design Data

Abstract

Single-case designs (SCDs) are an important source of evidence regarding the effects of interventions in certain areas of psychology, education, and the behavioral sciences. Researchers have traditionally used visual inspection of graphed outcome data to draw conclusions about intervention effects in single-case studies (Kratochwill, Levin, Horner, & Swoboda, 2014). However, there has been growing interest in using statistical methods to analyze data and estimate effect size indices from SCDs (Manolov & Moeyaert, 2017; Shadish, 2014). Statistical analysis potentially offers greater objectivity and transparency than visual inspection. However, all statistical methods entail assumptions about the data-generating process, and the validity of statistical inferences rests on whether these assumptions are realistic. Likewise, conducting Monte Carlo simulation studies to evaluate statistical techniques for single-case data requires simulation models that can produce artificial data with features similar to real empirical data.
The most common dependent variables in single-case research are measures of behavior, assessed through systematic direct observation (Gast, 2010). Commonly used procedures for systematic direct observation include frequency counting, continuous recording, momentary time sampling, and partial interval recording (Ayres & Gast, 2010). These procedures produce measurements in the form of rates or proportions. Crucially, behavioral observation data have features that are not well-described by regression models with normally distributed errors (Solomon, 2014; Solomon, Howard, & Stein, 2015), even though such models have been the predominant approach to statistical analysis of SCED data. Monte Carlo simulation studies investigating these models have also typically generated data from models with normally distributed errors.
The objective of this study is to examine the procedures used to collect behavioral outcome measurements, the features of the resulting baseline outcome data, and the relationships between measurement procedures and behavioral outcome data in SCED studies. We draw on a corpus of SCED studies from seven previously conducted systematic reviews (Barton, Stiff, Mauldin, & Choi, 2015; Dart, Collins, Klingbeil, & McKinley, 2014; Gage, Lewis, & Stichter, 2012; Ledford, King, Harbin, & Zimmerman, 2016; Maggin, Pustejovsky, & Johnson, 2017; Shogren et al., 2004; Verschuur, Didden, Lang, Sigafoos, & Huskens, 2014) covering a range of intervention classes and outcome constructs. The corpus includes over 300 studies and approximately 1800 cases. We have systematically coded characteristics of the measurement procedures used in included studies, including the class of outcome (e.g., disruptive behavior, pro-social behavior, communication), measurement system (including interval lengths), measurement metric, observation session length, and outcome scale range. Raw outcome data were extracted from graphs in the original source articles, in most cases by the authors of the contributing systematic reviews.
The poster will report the distribution of measurement characteristics, the distribution of baseline outcome data, and estimated statistical models that examine how measurement characteristics relate to features of the outcome distribution (e.g., how session length affects the degree of variability in baseline outcomes). These summaries will inform the development of more realistic assumptions regarding outcome distributions in SCED studies, as well as the design of future Monte Carlo simulation studies evaluating the performance of statistical analysis techniques for SCED data.

Authors

• James Eric Pustejovsky, The University of Texas at Austin

• Daniel Swan, University of Oregon

• Kyle Warren English, The University of Texas at Austin