Evaluating Methodological Quality of Single-Case Experimental Design Studies Using the What Works Clearinghouse Design and Evidence Standards

• In Event: Analysis and Meta-Analysis of Single-Case Experimental Data: Methodological Innovations and Complexities

Abstract

In 2009, What Works Clearinghouse (WWC) assembled a panel of recognized experts to draft design standards and evidence standards (Version 1.0 Pilot, for more information, see Kratochwill et al., 2010) for evaluating the effectiveness of a treatment using single-case experimental designs (SCEDs). Since then, many applied SCED researchers refer to Kratochwill et al. (2010) to design their study. In January 2020, the WWC released a Standard and a Procedure handbook (Version 4.1) in which it incorporated the SCED standards published earlier with few updates and changes. Some updates to the new version of the WWC Procedures and Standards Handbook are as follow: adding new procedures for calculating effect sizes (i.e., design-comparable effect size), adding requirement of providing raw data in graphical or tabular format), and removing “5-3-20” rule. The “5-3-20” rule was being applied for synthesizing SCED studies (at least five SCED studies conducted by at least three independent research team and at least twenty cases was needed to rate the effectiveness of the intervention).
Methodology. This study provides a comparison of the design standards for SCEDs published in 2010 with the recent ones published in 2020. More in particular, we are focusing on SCEDs published after 2010 that has been evaluated by WWC reviewers. Two independent researchers are going to review these studies by applying the new 2020 standards. We are especially interested whether the 2020 standards result in any differences in rating the studies. To do so, we have used the WWC Review Guide for SCED studies which can be found at WWC Website (https://ies.ed.gov/ncee/wwc/StudyReviewGuide).
Preliminary Results. Using the WWC website, we identified 27 SCED articles and doctoral dissertations published within the last 10 years. All these studies were reviewed by WWC reviewers implementing the old version of the standards. We have applied the new standards to three studies reviewed by the WWC reviewers with different ratings. The three studies rated so far did not show any discrepancy in their ratings with WWC reviewers’ ratings. The same procedure will be applied to review the remaining 24 studies and will be presented at the conference. This will allow us to make an in-depth analysis and comparison of the two versions of the SCED standards and discuss major implications for the field.
Significance. The WWC technical documentation (Kratochwill, 2010) is a frequently referenced and influential source in SCED studies for choosing suitable conditions (i.e., how many measurements and participants to be included in the SCED study). Considering the fact that new standards have just released in 2020, it is important to learn about the changes to the original document and how they can be implemented in future studies to gain more valid and reliable results from SCED studies.
 

Authors

• Marzieh Dehghan-Chaleshtori, University at Albany - SUNY

• Mariola Moeyaert, University at Albany

• Daniel Swan, University of Oregon