Innovations in Scholarly Publishing: Registered Reports for Hypothesis-Driven Research

• In Event: New Norms for Educational Research

Abstract

Registered Reports are a new form of empirical research article designed to improve the transparency and reproducibility of hypothesis-driven research. When authors initially submit a Registered Report, they submit their Introduction, focal hypotheses, and Methods section of a paper. Articles can then receive an “in principle acceptance” (or rejection) based on the quality of the question and methods rather than on the significance, direction, or effect size of the findings. This approach aligns editorial and author incentives towards publishing important questions with excellent research designs, while avoiding incentives that encourage publication of particular p-values or effect sizes, thereby mitigating publication bias. This presentation provides an overview of the benefits and challenges of Register Reports, along with a case study of implementing Registered Reports as a special topic within the journal AERA Open.

Benefits of Registered Reports
For studies with a clear hypothesis, the Registered Reports format has three key strengths compared with standard publishing.
1. First, it prevents publication bias (the “file-drawer” problem) by ensuring that editorial decisions are based on the theoretical importance and methodological rigor of a study, before research outcomes are known.
2. Second, by requiring authors to register their study methods and analysis plans in advance, it can reduce common forms of research bias including p-hacking and HARKing (hindsight bias) while still encouraging exploratory analyses.
3. Third, because protocols are accepted in advance of data being collected, the format provides greater incentive for researchers to conduct important replication studies and other novel, resource-intensive projects — projects that would otherwise be too risky to undertake if the possibility of being published were contingent on the results (Makel & Plucker, 2014).

Mechanics of Registered Reports
Registered Reports differ from the conventional publication process by having two phases.
In the first phase, authors write the Introduction and Methods section of a paper, before collecting and analyzing any data. Together with any prepared materials and analysis scripts, the manuscript is then reviewed by peers. High quality pre-registered protocols that meet strict editorial criteria are then offered in principle acceptance.
The second phase of peer review occurs after data collection, and resembles the regular peer review process. However, the in principle acceptance guarantees publication of the results provided authors adhere to their pre-registered protocol—regardless of the direction or effect size of the findings.
While Registered Reports are focused on testing pre-registered hypotheses, the final manuscript can certainly include exploratory analyses. Registered Reports allow a clear distinction between confirmatory (that is, pre-registered) hypotheses testing and exploratory analyses (see Gehlbach, et al., 2016).

Conclusion
Registered reports have begun to be published in cognitive psychology (Verkoeijen & Bouwmeester, 2014), educational psychology (Rowland, Bates, & DeLosh, 2014), and media psychology (Van der Zee, Admiraal, Paas, Saab, & Giesbers, 2017). To the extent that Registered Reports become established practices in education research, publication bias should decrease, p-hacking will be attenuated, and replication studies will become more common. As a result, authors of meta-analyses and policy-makers would gain much clarity around what works in education.

Authors

• Justin Fire Reich, Massachusetts Institute of Technology

• Tim van der Zee, Leiden University