Sensitivity Analyses Examining the What Works Clearinghouse’s Treatment of Overall and Differential Attrition

• In Event: Roundtable Session 97
  In Roundtable Session: The Magnitude and Direction of Effect Size Research

Abstract

The U.S. Department of Education, through its What Works Clearinghouse (WWC), has set thresholds for establishing when attrition can be expected to bias the impact estimates of randomized controlled trials. A key feature of this prior work was to set acceptable levels of expected bias resulting from attrition and perform a series of simulation analyses to ascertain when bias reaches unacceptable levels. The current work has managed to replicate these simulation studies. A second step entailed performing sensitivity analyses of how the amount and type of attrition, under varying assumptions of how much subjects’ likelihood of responding is related to their outcome, affects the bias estimates. The current study also examined how measurement error impacts the WWC’s attrition bias thresholds.

Authors

• Marsha S. Lewis, Ohio University

• John H. Hitchcock, Ohio University

• Gordon P. Brooks, Ohio University - Athens

• George A. Johanson, Ohio University