Multiple-Hypothesis Testing in Single-Case Experimental Designs

• In Event: 61.033 - Single-Case Experimental Designs: Developments in Statistical Analysis, Effect Size Metrics, and Meta-Analysis Methods

Abstract

Objectives. This presentation will address the issue of multiple hypothesis testing when significance tests are repeatedly implemented in single-case experimental designs (SCEDs) to make comparisons between pairs of design phases. This leads to an increased risk for making a Type 1 error, which is known as the family-wise error rate. Presenters will (a) point out typical SCED scenarios for multiple phase comparisons and multiple hypothesis tests, and (b) describe a procedure to control the family-wise error rate in SCEDs.

Theoretical framework. The multiple comparison problem occurs when a set of several hypothesis tests is conducted on the same data. In group designs, the more tests researchers apply to the same data, the greater the risk that an apparently significant difference is obtained by chance. This calls for procedures to control the family-wise error rate. In SCEDs this scenario occurs when researchers conduct non-parametric statistical tests for different phase comparisons. A researcher may compare baseline data to intervention, generalization, and maintenance phases and/or compare all four phases to one another, inflating the family-wise error rate.

Methods. Because data from SCEDs often do not meet assumptions for parametric data (e.g., distribution), traditional group design methods for analysis of variance and post-hoc correction (e.g., Bonferroni, Tukey) are difficult to apply. One of the alternatives is Dunn’s All Pair-wise Comparison for Unequal Group Sample Sizes (1964), which will be the focus since phases in SCED usually contain unequal amounts of data points. Multiple hypothesis tests using Dunn’s All Pair-wise Comparison will be performed on two different SCED scenarios. In addition, these scenarios will be analyzed by visual analysis using the standards of the What Works Clearinghouse (Kratochwill et al., 2010).

Data sources. Two data sets will be used: (a) one multiple baseline across behaviors investigating the effects of a phonological remediation treatment for skill acquisition and maintenance, which includes baseline, intervention, follow-up, and generalization phases (Rudolph & Wendt, 2014) and (b) a combination of multiple baseline and variation designs to teach requesting skills for communication, which includes five different sub-phases for incremental acquisition (Wendt et al., 2014). Statistical analysis will focus on pair-wise comparisons of baseline to subsequent intervention phases in order to pinpoint the most significant phase differences over the course of interventions, thus, illustrating the implementation of multiple-hypothesis testing while controlling the family-wise error rate.

Results. The results of multiple-hypothesis testing will be presented and compared to the visual analysis of both data sets. Advantages and disadvantages of this testing with respect to inflated error rate will be discussed. A SAS macro will be shared with attendees to allow for multiple hypothesis testing using their own data.

Significance. Failure to control family-wise error rates in SCEDs can have important real-world consequences: more phase comparisons are done on the same data, a significant difference is more likely to be found by chance. False inferences may be drawn on treatment efficacy solely on the basis of performing numerous tests and erroneously assuming treatment effectiveness.

Authors

• Oliver Wendt, Purdue University

• Bridget Miller, Purdue University