Open Science Research Practices: What, Why, and How

• In Event: Building a Culture of Open Science in the Education Community: Data on Adoption and a Platform for Change

Abstract

The objective of this presentation is providing an overview of open-science practices in the context of educational research, including a rationale for their use, their potential benefits, and identified obstacles and limitations. Open science is an umbrella term for practices aiming to increase the reproducibility, transparency, and accessibility of research by making scientific processes and products as open as possible (Stracke, 2020). Foundational scientific frameworks serve as the theoretical basis for the presentation. “Both Merton and Popper see science as a transparent and collective enterprise. Science should be open by default. In that regard, … ‘open science’ provides a convenient framework for what is already expected of science” (Fetcher & Ross-Hellauer, 2018, p. 1). In essence, research is theorized to be more reliable and impactful when it is open (Cook et al., 2018). Importantly, open science enables equitable access to the processes and products of educational research for all stakeholders (e.g., by making high-quality datasets open to all researchers and research reports available to stakeholders; Fleming et al., 2021).

Findings from large-scale reproducibility projects precipitated a reproducibility crisis in contemporary science (Baker, 2016). Although Makel and Plucker (2104) found that approximately 2/3 of replication studies in education reproduced previous findings, the rate of replication research is low in education (only 0.1% of articles), meaning the reproducibility of most findings in the research base is unknown. One factor potentially underlying the reproducibility crisis is the use of questionable research practices, such as p-hacking (i.e., trying multiple analyses until desired findings are reached), selective outcome reporting (i.e., only reporting desired outcomes), and hypothesizing after results are known (i.e., presenting hypotheses developed after analyzing data as a priori). Many education researchers have reported engaging in questionable research practices (Makel et al., 2021). Such threats to the validity and reproducibility of education research have important implications, as research is a basis for education policy and practice.

The primary focus of this presentation will be an overview of core open-science practices that education researchers are beginning to apply:
· Pre-registration: Publicly posting study plans and procedures before conducting a study (Gehlbach & Robinson, 2018)
· Registered reports: Submitting study plans to a journal for feedback and in-principle acceptance before the study is conducted (Reich et al., 2020)
· Open data: Making de-identified data publicly available for other researchers to reproduce study analyses, conduct robustness analyses, and examine novel research questions (van Dijk et al., 2021)
· Open code: Making analytic code publicly available to enable other researcher to reproduce study analyses
· Open materials: Making materials used in the study (e.g., survey instrument, intervention protocol) publicly available to facilitate replication
· Open access: Making research reports publicly available (e.g., by posting preprints on a print archive; Fleming & Cook, 2021)
For each open practice, I will present (a) how the practice works; (b) theorized benefits related to improved reliability, impact, and equitable accessibility of education research; and (c) potential limitations and obstacles (e.g., time and resources required, lack of clear norms in education research, primary focus on quantitative research).

Author

• Bryan G. Cook, University of Virginia