Meta-Analysis in Education: Past, Present, and Future

• In Event: Reconceptualizing Practical Meta-Analysis in Education

Abstract

As evidenced by the recent “look back” articles from Research Synthesis Methods (e.g., Chalmers, 2015; Glass, 2015; Rosenthal, 2015; Shadish & Lecy, 2015), the field of meta-analysis is reaching middle age. Nearly forty-years since its beginning, the field of meta-analysis has coalesced around a set of core methods. A variety of books (e.g., Lipsey & Wilson, 2001; Cooper, 2010; Hedges & Olkin, 1985) now exist on these methods, and workshops and courses are frequently taught at professional association meetings, online, and in graduate schools of education, social work, and public health.
The typical course or workshop on meta-analysis appears anchored to a historically-based sequence of topics, devoting little coverage to “advanced” methods and often leaving students to learn such techniques in less detail or on their own. This standard perspective is also visible in the production of contemporary syntheses in educational research, where researchers commonly focus on basic, univariate methods, saving “more advanced” methods for future research. This approach, however, is not one that is conceptually based. Students begin with fixed versus random effects discussions, despite the fact that most researchers implement random effects analyses. Similarly, the standard methods assume independent effect sizes – an assumption made out of statistical convenience in its early genesis – despite the fact that primary studies very frequently include multiple, correlated effects.
A historical approach to methods is not problematic in and of itself, insofar as similar progressions are found in many other parts of the applied statistics curriculum (e.g., the teaching of experimental design). However, in meta-analysis this historical approach approach muddles issues of data structures with those of model estimation. The standard approach is to begin with the methods and to fit the data to the methods – for example, choosing to use a “shifting units of analysis” approach or choosing a multi-level model – without a clear understanding of the underlying structure to the data and how that should guide analyses and interpretations. We will argue in this paper that, over the past decade, the development of methods for the handling dependent effect sizes and the development of freely available software packages together provide an opportunity for reimagining what the “standard” approach to the teaching and application of meta-analytic methods should be.
In this paper, we argue for a new framework for meta-analysis, one that begins with an understanding of the data structure, and then makes clear the distinction between this true structure and the assumptions required for estimation, and the conditions under which different estimation strategies perform well. The framework begins with an assumption of dependency, making clear to analysts that while there are many different choices available, there are trade-offs to each. Finally, we argue that meta-analysis is due for a mid-life crisis and the opportunity for integration and re-invigoration that naturally follows.

Authors

• Elizabeth Tipton, Teachers College, Columbia University

• Ariel M. Aloe, University of Iowa

• James Eric Pustejovsky, The University of Texas - Austin