Sampling Distribution of Non-Overlap Indices Using Bootstrapping Procedure: A Monte Carlo Simulation Study (Poster 3)

• In Event: Methodological Innovations and Complexities in the Analysis and Meta-Analysis of Single-Subject Experimental Design Data

Abstract

Different single-case experimental design (SCED) quantification measures require different inferential methods when making statistical inferences. Non-overlap statistics, one group of commonly used statistics in the field of SCED are challenging in making interpretation of their results and making inferences about the population. The main reason is that non-overlap statistics have inconsistent and unknown sampling distributions (under the alternative hypotheses that there is an intervention effect). Therefore, traditional statistical hypothesis testing cannot be applied to make inferences. Currently, the benchmarking method is most frequently used to make interpretations and inferences about intervention effectiveness. However, benchmarking scales are problematic because they are not context, data, and design-specific.
The purpose of this study is to (1) investigate the appropriateness of bootstrapping in patterning the sampling distributions of non-overlap statistics, (2) apply the bootstrapping method to establish Critical Intervals (i.e., the 95% interval range established by the empirical bootstrapping sampling distribution) of non-overlap statistics
A Monte Carlo simulation study was conducted to investigate the appropriateness of bootstrapping methods to pattern the sampling distribution of two non-overlap indices (i.e., PND and NAP). The manipulated design conditions were the intervention effect, within-case variance, and the number of measurement occasions. For each design condition, 1,000 datasets were generated. The specific data generation models and the design conditions will be presented during the poster presentation.
The results of the simulation study indicate that the bootstrap method can be applied under certain conditions to NAP to make inferences about intervention effectiveness and can potentially be applied to other “complete” non-overlap indices, such as Tau and PAND. In contrast, it is not recommended to be applied to PND and other similar “incomplete” non-overlap indices such as PEM. More specifically, the bootstrap method and bootstrap confidence interval are recommended when the intervention is small to medium (and the within-case variance is around the same size of the intervention effect). Researchers are also encouraged to include more measurement occasions to reduce Type I and Type II errors.
Bootstrapping is a generic method that can be applied to different statistics and data. It is promising in patterning the sampling distribution of some non-overlap indices (i.e., NAP and Tau-UAvsB). The Critical Interval is design, context and data-specific and can be used to make accurate and consistent conclusions about the intervention effect. Thus, researchers are encouraged to use Critical Intervals instead of benchmarks in making inferences about non-overlap indices.

Authors

• Yukang Xue, University at Albany - SUNY

• Xinyun Xu, University at Albany - SUNY

• Mariola Moeyaert, University at Albany - SUNY

• Wim Van den Noortgate, KU Leuven