Preregistration and Transparency Into the Research Process

• In Event: Actions for Increasing the Credibility of Educational Research

Abstract

By submitting a detailed study and analysis plan to a registry prior to conducting a study, scientists can increase transparency into the research lifecycle while making a better distinction between planned, confirmatory hypothesis tests and unplanned, exploratory research. Both modes of research are important. However, without a clear distinction, one can unintentionally present results of exploratory analyses using statistical tools designed for a-priori hypothesis testing, which invalidates their diagnostic value. Furthermore, it presents results as being more surprising and publishable, at the expense of their credibility.

Preregistration is appropriate for hypothesis testing studies and for those where methodological decisions, such as variable construction or inclusion rules, can influence inferences. While some study designs present particular challenges for preregistration, such as longitudinal studies or those that rely on existing datasets, the benefits of transparency, better planning, and increased clarity between confirmation and discovery still adds value to the research.

Additional measures to increase the evidential value of reported findings include data, code, and materials sharing, so that the complete research lifecycle can be better evaluated. This transparency increases the impact of reported findings, their credibility, and is particularly useful for original authors, who are better able to quickly adapt well curated materials for their own reuse.

Without preregistration, the scientific literature is prone to bias and exaggeration. The bias for statistically significant research findings is well known (Dwan, Gamble, Williamson, Kirkham, & the Reporting Bias Group, 2013), but it is also true that this bias produces a scientific literature that is unrepresentative of the total credible evidence. Franco, Malhotra, and Simonovits (2014) found that published research results were unrepresentative of the entire collected evidence even when null results were generated using common methodology. Preregistration can help researchers add credibility to their results, which are more likely to present credible null results (Kaplan & Irvin, 2015) or more modest effect sizes than when preregistration is not used (Dechartres, et al., 2016; O’Boyle, Banks, & Gonzalez-Mulé, 2017; Schäfer and Schwarz, 2019).

However, preregistration is no magic bullet. Preliminary evidence by Clasen and colleagues (2019) found that the first 27 preregistered studies published in Psychological Science deviated from their analysis plans without complete disclosure. This mirrors lessons learned from clinical trials, where outcome switching still occurs (Goldacre et al., 2019).

But even without complete adherence to preregistered plans, the act of preregistration increases transparency by making discoverable these shortcomings that would otherwise be impossible to detect. The onus is on the research community to gain experience with the process of preregistration and complete reporting. Finally, reviewers and authors must come to expect complete reporting of outcomes by not penalizing transparent changes to research designs. This shift in expectation must occur in tandem with increased expectation for preregistration when making inferential claims.

This session will introduce preregistration as a method as well as share some of the research cited above regarding impact. Resources for preregistering studies will be shared as will be suggestions for how to respond to common concerns or barriers.

Author

• David Mellor, Center for Open Science