The Joint Influence of Convention and Evidence Among Education Researchers

• In Event: 25.015 - Improving the Evaluation and Scalability of Social Psychological Interventions in Schools

Abstract

Education researchers constantly advocate for greater use of evidence-based practices over conventional practices that lack evidence to support their efficacy (see Tseng, 2012; Wilson, 2011). These researchers exert considerable influence in the research-to-practice pipeline through their selected topics of research, advocacy for particular kinds of educational reform, roles as journal editors and reviewers--which put them in a position of gatekeeper as to what work is publicized--, and roles on grant committees that decide what research will have the potential for more rigorous testing and greater scaling. Given their important role in the research-to-practice pipeline, we thought it would be important to ask to what extent these researchers themselves base their support for educational interventions on evidence rather than convention.

To explore this question we recruited over 1500 educational researchers attending American educational research conferences to participate in a randomized field experiment. Our study was fundamentally a two-by-two design where participants were randomly assigned to review an educational intervention that was either conventional or unconventional in its implementation methods, and was supported by relatively strong or weak evidence in its evaluation. The conventional program targeted racial achievement gaps through a comprehensive overhaul of the school’s curriculum and community outreach programs at a cost of $2500 per student. The unconventional program targeted racial achievement gaps through a series of one-hour psychological interventions that cultivated student beliefs about social belonging, their ability to grow, and their personal values at the cost of $20 per student. The evidence supporting the efficacy of these programs was either relatively strong (a multi-site, 3.5 year RCT) or weak (anecdotal assessments from teachers and students).

As our principal outcomes, we measured our researcher-participants’ support for the intervention’s broader implementation and their own assessment of the quality of evidence supporting the program. Strong and weak evidence were consistently rated as such regardless of the conventionality of the program they supported (Fig. 2). However, these assessments of evidence quality did not equally affect participants’ support for the programs. A program’s conventionality increased support as much as the evidence quality supporting the program’s efficacy. As a result, conventional programs with weak evidence received equal support to unconventional programs with strong evidence (Fig.1). This pattern occurred even among participants who had completed their doctorate. This is troubling, because it suggests that even with advanced training, researchers are still assessing new educational interventions as much for their conventionality as for the evidence behind them.

However, this dual influence of conventionality and evidence quality was not immoveable. Later studies found two ways of bringing support levels for unconventional programs to the same level as conventional programs when both had strong evidence. One method was to have participants precommit to scientific evaluation standards before viewing any program details. The other method involved changing the recipient of the unconventional program from students to teachers (Fig. 3). Our discussion focuses on steps that our field can take to increase support for promising evidence-based interventions that do not rely on conventional techniques.

Authors

• Joseph Thomas Powers, Stanford University

• David Yeager, The University of Texas - Austin

• Geoffrey L. Cohen, Stanford University