Registered Reports for Hypothesis-Driven Research

• In Event: Actions for Increasing the Credibility of Educational Research

Abstract

What are Registered Reports?
Registered Reports are an approach to conducting and publishing empirical research. This process aspires to improve the transparency and reproducibility of hypothesis-driven research. In a Registered Report, authors first submit their Introduction, focal hypotheses, and Methods. This partial manuscript is sent out for peer review. At this point, a piece might receive a rejection, a revise and resubmit, or an “in principle acceptance.” Once authors receive this in principle acceptance, the study is conducted. After the study’s completion the full manuscript is submitted. Assuming no deviations from the plan outlined in the methods section and/or in the preregistration (Gehlbach & Robinson, 2018), the paper is accepted. Thus, the acceptance or rejection of scholarly work is based on the quality of the question and methods rather than on the direction or effect size of the findings.

Why Does Educational Research Need Registered Reports?
For studies with a clear hypothesis, the Registered Report format has a number of benefits for researchers and the field more broadly. First, Registered Reports have been shown to reduce publication bias. In a 2019 study, Allen and Mehler found that across 113 published registered reports in the fields of biomedicine and psychology, 55% failed to find support for their a priori hypotheses. This is far higher than the estimated 5% to 20% of traditional articles that result in null findings. Second, the format also appears to reduce questionable research practices by authors (Wiesman, Watt, & Kornbrot, 2019). By requiring authors to register their methods and analysis plans in advance, there are fewer opportunities for research bias including p-hacking and HARKing (the hindsight bias that is enacted by “hypothesizing after results are known”) while still welcoming unregistered analyses that are clearly labelled as exploratory. Third, the Registered Report approach aligns editorial and author incentives towards publishing important questions with excellent research designs. Protocols are often accepted in advance of data being collected so it mitigates the risk of pursuing risky, novel projects (i.e., projects where the publishability of the outcome is contingent on the results, such as replication studies or resource-intensive projects) (Makel & Plucker, 2014).

Objectives & Significance
We explore the theoretical and practical implications for introducing Registered Reports as an accepted and common form of empirical work in education research. The specific objectives of this session are to:
Introduce the process for submitting a Registered Report in education;
Consider the advantages and disadvantages of this type of empirical research article for educational studies;
Provide lessons learned and best practices for submitting Registered Reports; and
Discuss implications for the field of education research, more broadly.

To make each of these four objectives more accessible to the audience, each will be discussed from the perspective of a reviewing editor as well as a submitting author.

Establishing Registered Reports as a normal education research practice would result in the publication of a much greater number of null findings. In turn this would empower researchers and policymakers with a much better grasp on what works in education.

Authors

• Carly D Robinson, Brown University

• Hunter Gehlbach, Johns Hopkins University