Correspondence Metrics for Assessing Replication Success of Causal Effect Estimates

• In Event: Educational Research From Different Lenses

Abstract

Although replication is a popular topic in social sciences research, there are few concrete recommendations on the best statistical methods to determine if a replication effort is successful. This study investigates several popular methods for assessing replication success by deriving formulas to ascertain each method's probability of showing replication success. These methods are also applied to an analysis of existing replication data from the Open Science Collaboration (2015). Planning replication studies using these methods is also briefly discussed. Overall, methods that focus on determining if both studies lead to the same substantive conclusion have undesirable properties that can be avoided by using alternative methods that test for both the difference and significance of effect estimates.

Authors

• Patrick Sheehan, University of Maryland - College Park

• Peter M. Steiner, University of Maryland - College Park