The Implications of New Norms in Education Research for Research Synthesis

• In Event: New Norms for Educational Research

Abstract

Extensive debate and criticism of common, questionable practices within social and health sciences has emerged over the last decade (e.g., Gelman & Loken, 2014; Ioannidis, 2005; Simmons et al., 2011). The main issue is that many individual research results and broader assessments of the state of a literature, may be illusory. Putting aside cases of blatant fabrication, these illusory findings and conclusions can emerge from common practices that lead to publication bias (e.g., Rothstein, Sutton, & Borenstein, 2005) and other biases related to researcher decision making, including selective outcome reporting, selective analysis reporting, significance chasing or p-hacking, and inadequate treatment fidelity information (e.g., Hulleman, Murrah & Kosovich, this session; Robinson & Gehlbach, this session; Ioannidis, 2005). The first purpose of the current talk is to discuss the implications of problematic practices and biases within primary research for research syntheses, as well as the implications of potential solutions.
It has been argued that recommendations for practice and policy should rely primarily on research syntheses rather than individual research studies (e.g., Schraw & Patall, 2014). Given the critical role that research synthesis plays in various forms of decision-making, it is alarming to note that problematic practices in primary research will transfer directly to biases in research syntheses. Moreover, undesirable consequences of such biases (e.g., inaccurate knowledge base, hampered research progress, misuse of research resources, misguided practice and policy making, climate of mistrust) are likely magnified by research syntheses.
Several strategies, if widely adopted, may help to mitigate the production and communication of biased research conclusions, including pre-registration of studies, registered reports, and rigorous, systematic techniques to assess treatment fidelity (e.g., Robinson & Gehlbach, this session; Reich & van der Zee, this session; Hulleman, Murrah & Kosovich, this session). Such practices stand to substantially improve the quality of information produced by research syntheses and address some of the most significant threats to the validity of research synthesis. Notably, such strategies stand to help address 1) publication bias and difficulties associated with identifying and collecting unpublished literature (including researcher reluctance to share unpublished information), 2) a lack of comparable replications, 3) difficulties associated with retrieving comparable effect sizes, and 4) a lack of opportunity or information needed to explore design characteristics that explain variance in effects.
The final purpose of the talk is to highlight parallel biases and solutions that stem from research synthesis practices.
• Syntheses are also vulnerable to biases that result from researcher decision-making.
o However, research syntheses could and should also be pre-registered.
• Second, research syntheses that illustrate results that run counter to current expert or public opinion are sometimes suppressed and often denigrated or devalued.
o However, registered reports could and should be used for publication decisions for research syntheses.
• Finally, researchers should move away from research synthesis as a retrospective approach to research synthesis as prospective approach by designing studies with the intent to contribute to research syntheses and by planning research syntheses prior to and as a guide for primary research (e.g., Ioannidis, 2010).

Author

• Erika Alisha Patall, University of Southern California