Mitigating Performance-Enhanced Data Through Preregistration in Education

• In Event: New Norms for Educational Research

Abstract

Objectives & Framework
Like performance enhancing drugs inflating apparent athletic achievements, several common social science practices produce illusory results. Multiple techniques allow researchers to discover “significant” findings that are actually artifacts of study design, analytic approach, and/or reporting decisions.
The problem of illusory results in education parallels many other disciplines. Consequently, the research base in our field contains many erroneous “facts” (Ioannidis, 2005). This presentation addresses one solution for preventing illusory results: pre-registration plans. We build upon the natural- and social-sciences’ early pre-registration endeavors to explore how education might systematically enact pre-registrations to improve the scientific process, while accounting for the unusual breadth of methodological and philosophical approaches to doing research in the field.

Why Do We Need Pre-Registration?
During the study design, analyses, and reporting phases researchers can generate more or fewer decision points in their study. As researchers make decisions that enhance their odds of finding something scientifically interesting and potentially publishable, these choices create numerous “researcher degrees of freedom,” and make it hard for colleagues in the scientific community to disambiguate real findings from chance occurrences (Simmons et al., 2011). In the absence of multiple replication studies, detecting which particular findings are illusory and which are real is essentially impossible. Currently, the best we can do is to identify researcher practices that, on average and in combination, are likely to produce illusory findings.

What is Pre-Registration?
To mitigate the likelihood of researchers unwittingly generating illusory results, other disciplines have begun employing pre-registration plans. A pre-registration plan describes researchers’ exact plans with respect to study design and analysis. The plan is then posted to public repository before the study is conducted or before the data are analyzed. By specifying the details of the study ahead of time, the reporting of findings becomes largely straight-forward (i.e., report on the research questions and hypotheses that were described in the pre-registration using the pre-specified analytic approach).
Philosophically, two key tenets drive the logic behind pre-registrations. First, pre-registrations facilitate researcher transparency with respect to study design, data analysis, and reporting choices. Second, pre-registering studies should remove researchers’ degrees of freedom. In other words, pre-registration combats the natural human inclination to craft a post-hoc story and eliminates the numerous decision forks that ultimately result in inadvertent p-hacking.
Table 2 presents some preliminary guidelines for the research community to test out to see whether a set of new norms can improve the efficacy of educational research.


Conclusions & Significance
Within and beyond the academy, the consequences of illusory results can be immensely demotivating. Even for scholars who publish only work of unimpeachable integrity, some portion of their knowledge base comes from other scientists. Likewise, when practitioners and policy-makers cannot disentangle genuine findings from fake, they may rely on bad research or their gut intuition guide their decision-making. Instituting a norm of pre-registering studies in education could mitigate these problems substantially. We argue that Division C scholars are ideally positioned to begin adopting this new norm.

Authors

• Carly Dolan Robinson, Harvard University

• Hunter Gehlbach, University of California - Santa Barbara