How to Evaluate Journal Implementation of Open Science Standards: The TRUST Process

• In Event: How to Evaluate Journal Implementation of Open Science Standards: The TRUST Process

Abstract

This demonstration aims to introduce a process for assessing the implementation of open science standards at journals in the educational, social, and behavioral sciences. Our "TRUST Process" involves three new instruments for rating journal instructions to authors (policies), manuscript submission systems (procedures), and published articles (practices) according to standards in the Transparency and Openness Promotion (TOP) Guidelines. The tutorial will begin with an overview of the motivation for open science standards: i.e., to mitigate detrimental research practices associated with closed science. Next, we will describe how we developed the TRUST Process and its tools for assessing journal implementation of open science standards (including coherence of implementation across journal policies, procedures, and practices). As a demonstration of the TRUST Process, we will then present (i) results from an assessment of 340 journals that have published research used to inform evidence-based practice, and (ii) how the TRUST Process was used to update open science standards at an education journal (Journal of Research on Educational Effectiveness). We will conclude with a live demonstration of the TRUST Process with a journal nominated by an audience member. Attendees will learn how to assess implementation of open science standards at education journals and how to use the TRUST Process for continuous quality improvement efforts at journals for which they serve on the editorial board.

Regarding JREE, I was tasked in my role as Methodological Transparency Editor with updating journal standards related to open science. I used the above process and instruments to assess our journal’s previous policies, procedures, and practices. Results from this exercise informed iterative conversations among the editorial board about our instructions to authors, manuscript submission system, and journal article template. The resulting documents are all open access and reusable with a CC0 license (https://osf.io/ckuqy/):
- Updated Instructions to Authors to be aligned with TOP Level 1: https://osf.io/kt4sz/
- To ensure our procedures are commensurate with our new policy, we now ask authors during manuscript submission to disclose whether or not they used open research practices via a TOP Guidelines Disclosure Form: https://osf.io/tuhca/. We also created reporting checklists for RCTs based on CONSORT-SPI (https://osf.io/yu6w4/) and systematic reviews based on PRISMA (https://osf.io/c24k9/).
- To ensure our practices are commensurate with our new procedures, we updated our article template to include a section on “Open Research Statements” that systematically includes the information reported during manuscript submission. Here’s an example showing these statements (which are visible even without a journal subscription): https://www.tandfonline.com/doi/full/10.1080/19345747.2022.2074929
- To help our authors meet these standards, we created a list of curated resources on the open research practices in the TOP Guidelines: https://www.sree.org/research-transparency

All materials related to the TRUST Process and its demonstrations will be available open access for the audience to use and share freely:
- TRUST Process: https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-021-00112-8
- Tool for assessing policies in their instructions to authors: https://osf.io/56pgq/
- Tool for assessing procedures in their manuscript submission systems: https://osf.io/u3dak/
- Tool for assessing practices in their published articles: https://osf.io/ctv9q/

Author

• Sean Grant, University of Oregon
  Presenting Author