A Comparison of Non-Regression-Based Effect Size Metrics for Single-Case Experimental Designs

• In Event: 61.033 - Single-Case Experimental Designs: Developments in Statistical Analysis, Effect Size Metrics, and Meta-Analysis Methods

Abstract

Objectives. This presentation will address current methodological issues related to non-parametric effect size estimates in single-subject experimental designs (SCEDs).

Theoretical Framework. Controversy exists as to which techniques are most appropriate to analyze between-phase differences in SCEDs and derive meaningful effect size estimates. Two general strategies have been proposed. First, regression approaches assess efficacy by using linear-regression techniques to model repeated observations. Recent developments include a generalized least squares regression approach (Maggin et al., 2011), and multilevel modeling (van den Noortgate & Onghena, 2008). These may allow a fine-grained analysis of effects on level and slope, and an analysis of overall treatment effect sensitive to inter- and intra-subject variability. Non-regression approaches use the amount of non-overlapping data as indicator of performance differences; the extent to which data in baseline versus intervention phases do not overlap indicates magnitude of effect. These procedures are not impacted by data assumptions of normal distribution, equal variance, and serial independence (which often are not met by SCED data). Their disadvantage is the inability to describe trend and variability, and being insensitive to the magnitude of mean level shift.

Methods. The family of non-overlap metrics includes the established Percentage of Non-overlapping Data (PND; Scruggs, Mastropieri, & Casto, 1987) in addition to newer metrics such as the Improvement Rate Difference (IRD; Parker, Vannest & Brown, 2007), Non-overlap of All Pairs index (NAP; Parker & Vannest, 2009), Pairwise Data Overlap Index (PDO2; Parker & Vannest, 2007), Percentage of All Non-overlapping Data (PAND; Parker et al., 2007), Percentage of Data Exceeding a Median Trend (PEM-T; Wolery, Busick, Reichow, & Barton, 2010), Percentage of Data Points Exceeding the Median (PEM; Ma, 2006), Percentage of Non-overlapping Corrected Data (PNCD, Manolov & Solanas, 2009), and Tau-U (Parker et al., 2011). A field test is being conducted on these metrics to compare their performances in detecting and accurately representing intervention effects. A uniform probability plot will compare the nine metrics statistically and show their ability to discriminate low, moderate, and strong effects. An intercorrelation matrix for the metrics will demonstrate how closely or loosely these are related. A comparison to visual analysis of the data set will reveal how well each metric corresponds with visual judgments.

Data sources. All ES metrics will be applied to the same set of 64 published SCED studies taken from a recent meta-analysis of autism intervention research (Wendt, 2009). These 64 experiments contain a total of 140 participants, 10 different single-subject research designs, allowing for 292 instances of effect size calculation.

Results. Performance differences, advantages and disadvantages of each technique and the relationship among the different metrics will be illustrated. Recommendations will be derived for applied researchers planning to implement a metric for individual SCEDs and their syntheses.

Scientific significance. An urgent need exists for identifying reliable statistical procedures for effect size calculations with SCEDs. Broad movements towards “evidence-based practices, interventions or treatments” are putting SCED research under pressure to derive more rigorous techniques for documenting the magnitude of treatment effect.

Authors

• Oliver Wendt, Purdue University

• Anya S. Evmenova, George Mason University