Methodologist: Research Design Stage

• In Event: Open Science 201: Research Life Cycle From Various Stakeholder Perspectives

Abstract

This panelist will talk from the perspective of a research methodologist, discussing how open science practices are relevant to study design and data analysis. Particular emphasis will be placed on three areas: 1.) discussing the application of open science principles when using modeling techniques (such as latent variable, mixture, and multilevel modeling) 2.) the importance of deemphasizing default tests of meaningless null hypotheses (i.e.- does anyone care about whether a reported correlation coefficient is statistically different from 0?), and 3.) The importance of reproducibility of results, and the skills, habits, and behaviors required to create reproducible results.

Attendees will leave the session ready to apply open science principles to conduct research using exploratory and confirmatory modeling techniques.
A. Relevant Terms (in addition to relevant terms related to open science-- such as preregistration)
Exploratory and Confirmatory Research
Statistical significance, null hypothesis testing, effect sizes
Reproducibility
Error variance , Variability

B. Background Information
In this presentation, I will discuss my personal experiences-
Teaching Open Science principles as part of a PhD level research methods class
Managing the design and analysis of large research projects, especially with an eye toward reproducibility of results
Serving as a research methodologist on projects, which often entails serving as the “conscience” of the project

C. Expected Challenges and Barriers

It is much harder to be thoughtful about analyses than to revert to simplistic rules (such as statistically significant or not).
Training-- we need to change the way that we train future researchers
There is often a tension between technical and conceptual aspects of methods-related research.

D. Tips of Trade

Preregistration is a boon for methodologists. For far too long, researchers have “designed” studies and collected data without thoughtfully planning their analyses. Preregistration is an opportunity for us to be more thoughtful and planful in our research, and it forces people to think through their analyses prior to conducting the study. Tip: Many elements of preregistration appear in IRB applications, and many IRB applications would be enhanced by including components of the pregistration. Begin the preregistration in tandem with the IRB.
Develop a reproducible workflow. In addition, document not just the decisions that you make as part of the research process, but also the reasons for making those decisions. Ideally, the “how” from raw data to the end product should be automated, and the “why” should be fully documented.
View analyses from the vantage point of a lawyer, rather than an accountant. We are weighing the evidence that supports and refutes the hypothesis. Relatedly, the world is not black and white. Avoid false dichotomies and recognize the importance of measurement error.

Author

• D. Betsy Mccoach, University of Connecticut