Paper Summary
Share...

Direct link:

Effects of Measurement Operations on the Magnitude of Nonoverlap Effect Sizes for Single-Case Experimental Designs

Sun, April 19, 10:35am to 12:05pm, Sheraton, Floor: Ballroom Level, Sheraton IV

Abstract

Non-overlap statistics are the most widely used category of effect size measures in meta-analysis of single-case research (Maggin, O’Keeffe, & Johnson, 2011; Smith, 2012). This family of effect sizes includes the percentage of non-overlapping data (PND; Scruggs, Mastropieri, & Casto, 1987), the percentage of all non-overlapping data (PAND; Parker, Hagan-Burke, & Vannest, 2007), the non-overlap of all pairs (NAP; Parker & Vannest, 2009), and a number of others. These measures are sometimes described as “non-parametric” effect sizes, in that they are not premised on assumptions regarding the normality of outcome measures and are insensitive to outliers (cf. Parker, Vannest, & Davis, 2011). These properties are seen as advantageous because many outcome measures used in single-case research are not well-modeled by normal distributions. However, little previous research has examined the characteristics of non-overlap effect sizes under data-generating models that are more plausible for the types of outcome measures used in single-case research. The present study aims to fill that gap, by studying the behavior of non-overlap effect sizes using data simulated from a realistic model for direct obervation of behavior.

Behavioral measures derived from direct observation are by far the most common type of outcomes in single-case research (Gast, 2010). A variety of different procedures are used to record direct observation of behavior, including continuous recording, momentary time sampling, frequency counting, and partial interval recording. Each of these procedures can be used for shorter or longer periods of observation, and the number of observation sessions per phase may also vary from study to study. If an effect size is to be useful for purposes of meta-analysis of single-case studies, its magnitude must be relatively insensitive to such details, because they are incidental to the empirical phenomenon under study. Effect sizes metrics that are sensitive to measurement operations will create problems for meta-analysis by obscuring substantive variation in study results and reducing the interpretability of the findings.

I evaluate the effect of measurement operations on the expected magnitude of the non-overlap statistics through a simulation study. To generate realistic behavioral observation data, I employ an alternating renewal process model (Pustejovsky, 2014; Rogosa & Ghandour, 1991), special cases of which have been used in many previous simulation studies of behavioral observation procedures (Pustejovsky & Runyon, 2014). The following factors are varied: recording procedure, interval length (for momentary time sampling and partial interval recording only), length of observation session, number of observations in the baseline phase, and number of observations in the treatment phase. Results indicate that all non-overlap statistics are sensitive to the length of the observation session and the choice of recording procedure. Furthermore, some are highly (PND) or moderately (PAND) sensitive to the length of the baseline phase. In light of these sensitivities as well as other criticisms of the non-overlap measures (Shadish, Rindskopf, & Hedges, 2008; Wolery, Busick, Reichow, & Barton, 2010), I conclude that non-overlap statistics are unsuitable for use in meta-analysis of single-case research when the outcomes of interest are measured through direct observation of behavior.

Author