Minimizing Research and Data Waste in Education

• In Event: Reconceptualizing Practical Meta-Analysis in Education

Abstract

The term meta-analysis was coined in education by Gene Glass to refer to ‘‘the statistical analysis of a large collection of analysis results from individual studies for the purposes of integrating findings’’ (Glass, 1976, p. 3). Since then, meta-analysis has become an important tool to inform decision-makers in many disciplines. Although, arguable, meta-analysis is mostly known as a set of statistical tools to combine effect sizes, meta-analysis techniques have developed and its usages extending beyond reporting the overall effect of an intervention. The purpose of the present paper is to describe two alternative applications of meta-analysis to educational research. We will demonstrate the benefits and potential limitations of two alternative applications via examples.
We first describe applications of meta-analysis techniques for informing research priorities. Meta-analysis techniques have been proposed and utilized to increase the value of research and reduce research waste in health disciplines (Chalmers et al., 2014). Methods for identifying what is already known (and not known) in a particular area can be used to identify research priorities. Similarly, when further evidence is needed in a specific area, meta-analysis of relevant studies can be used to plan target number and size of future studies (Roloff, Higgins, & Sutton, 2013). We describe and demonstrate how educational research can benefit by using meta-analysis techniques in setting research priorities by using the research on the effect of summer schooling as example.
Second, we describe applications of meta-analysis techniques for making better use of existing data. It is common for states to share students’ performance data and school personal data at some aggregated level. While aggregation facilitates the ability of states to share their data, it presents its own challenges as well. For example, the level of attempted inference must align with the level of data presented. Moreover, if data are analyzed using standard statistical techniques and software packages, within-group variability is typically ignored. Ignoring within-group variability leads to lost information and potentially misinterpreted coefficients. Fortunately, educational researchers do not have to ignore the within-group variability, nor must they rely on typically restricted data files to conduct proper analyses. We illustrate how data analysts can instead rely on the special case of mixed models analysis which allows for the treats within-group variances as known. This special case of mixed models is refer to as v-known variance in Raudenbush and Bryk (2002) and it is also the statistical approach used in meta-analysis (Hedges & Olkin, 1985) to combine effect sizes. An extension of this model using propensity scores will be introduced.

Authors

• Ariel M. Aloe, University of Iowa

• Elizabeth Tipton, Teachers College, Columbia University

• Adam Reeger, University of Iowa

• Rabia Karatoprak Ersen, The University of Iowa

• Seohee Park, The University of Iowa