Assessing Meta-Analysis Results From Regression Models of Different Sizes

• In Event: Meta-Analysis of Complex Primary Studies

Abstract

Several methods for synthesizing results of OLS regression models have been proposed over the past decade (e.g., Aloe & Becker, 2012; Keef & Roberts, 2004; Peterson & Brown, 2005; Wu & Becker, 2013). One of these approaches involves synthesizing partial effect sizes extracted from regression models (Aloe & Becker, 2012; Aloe & Thompson, 2013). The collection of regression models from studies in a meta-analysis will rarely be identical because primary researchers change the numbers and operationalizations of predictors, as well as other features which impact the value and interpretation of partial effects. This means that partial effects with different sets of variables controlled will likely arise across these diverse models.
However, it is not clear how typical meta-analysis summaries behave when indices reflecting a mix of different models are combined. Thompson and Aloe examined the behavior of three partial effect sizes – the partial correlation, semi-partial correlation, and beta weight – from regression analyses where some studies do not examine the full set of predictors that form the most complete model. That is, across studies, the sizes of models vary. A simulation study examined the impact of synthesizing effects from such mixed collections of regression models on standard meta-analytic (Hedges & Olkin, 1985) quantities. Inter-predictor and predictor-outcome correlations were varied to create one set of models with poor fit, and a second set of models with adequate fit, as determined by the overall R^2 value for each model. The number of studies in the pseudo meta-analyses was 5, 10, 20, 30, or 40. Specifically, the homogeneity measures Q, I^2, and τ^2, were computed, as well as the random-effects means and their variances, across 10,000 replications.
Overall, results suggest the effect of combining regression results from models of varying size has less detrimental impact when the primary studies show poor model fit (i.e., low variance explained) compared to when primary studies show better model fit. Specifically, for models with better fit, overall weighted average estimates deviated from the full-model (true) values moreso than weighted averages for models with poorer fit. Values of I^2 were often large (e.g., I^2 > 60%) for models with better fit and substantially smaller (e.g., I^2 < 15%) for models with poor fit. Meta-analyses of regression results should distinguish between statistical heterogeneity and variation that arises due to the fact that different variables appear in the synthesized models. Last, there were minimal differences in the behavior of the three different partial effect sizes considered in this paper. Thus, the decision about which partial effect size to use should be made by taking into account the research questions, the interpretations that the meta-analyst wishes to achieve, and what information is reported in the primary studies.

Authors

• Christopher Glen Thompson, Florida State University

• Ariel M. Aloe, University of Iowa