Quality Appraisal of Single-Case Experimental Designs: Current State of the Art

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Innovations and Complexities

Abstract

This presentation will contrast different tools for the appraisal of methodological quality of single-case experimental designs (SCEDs). Movements over the last decade towards evidence-based practice (EBP) have brought more attention to the task of critically appraising SCEDs to distinguish high quality from low quality studies.
Consumers of SCED research need to evaluate the methodological rigor of any SCED investigation before relying on study results for EBP purposes. Similarly, applied researchers aiming to synthesize SCEDs in a systematic review have to assess study quality and assign more weight to high quality studies. This process of examining internal and external validity of a published research report is called critical appraisal. Checklists and scales containing rigorous sets of criteria against which a research report is evaluated are known as critical appraisal tools. Such tools are common practice in the appraisal of group designs; for SCEDs however, quality appraisal is still in its infancy.
Wendt and Miller (2012) compared seven quality appraisal tools with respect to their compliance with current standards for conducting single-case experiments (Horner et al., 2005) as well as their performance in evaluating SCED quality. Considerable variability was noted relative to construction and content of the tools, which consequently led to variability in their evaluation results. Few tools provided empirical support for the validity of item construction and reliability of use. The Evaluative Method, the Certainty Framework, the What Works Clearinghouse (WWC) Standards, and the Evidence in Augmentative and Alternative Communication (EVIDAAC) Scales were identified as the more suitable appraisal instruments currently available.
Recently, newer appraisal tools have emerged including the Protocol for Assessing Single-Subject Research Quality (PASS-RQ; Maggin et al., 2014), the Single Case Analysis and Review Framework (SCARF, Ledford et al., 2016), the Scientific Merit Rating Scale (National Autism Center, 2015), and the rubric of The National Professional Development Center on Autism Spectrum Disorders (Steinbrenner et al., 2020). The proposed presentation will be an update to the Wendt and Miller (2012) review. The four newer tools will be compared to the four instruments that turned out suitable in the earlier comparison. In particular, goals are to (a) introduce the currently available appraisal tools and summarize their defining features, (b) compare these tools to a current standard of research design for SCEDs (i.e., Horner et al., 2005), (c) use the tools across a variety of different SCEDs to investigate their performance in differentiating study quality, and (d) reveal directions for the further development of SCED quality appraisal tools as well as recommendations for researchers and practitioners.
Previous results suggested the Evaluative Method might be best suited for comprehensive systematic reviews across design methodologies. The WWC Standards appeared suitable when aiming for a thorough assessment of internal validity. The EVIDAAC Scales seemed useful when critically appraising comparative treatment designs. The update will help to clarify if the newer tools add any further advantages. Such results will contribute towards a more standardized and valid framework for the critical appraisal of SCEDs as a common “gold standard” is yet to be identified.

Authors

• Oliver Wendt, University of Potsdam

• Bridget T. Miller, University of South Carolina - Columbia