Dependence of Slopes From a Single Sample

• In Event: Meta-Analysis of Complex Primary Studies

Abstract

Typical univariate meta-analysis techniques assume that the effects being summarized are independent of each other. When all indices arise from different samples, this is easily achieved. However, it is not uncommon for studies to present more than a single effect size.
The effects of the dependence due to multiple outcomes or multiple time points have been well documented (e.g., Becker, 2000; Gleser & Olkin, 2009). One less understood form of dependence is examined in “Dependence among slopes from a single sample” by Becker, Olkin and Aloe. They consider the practice, common in some domains, of deriving effect sizes from all competing models estimated for a single sample, and evaluate the level of dependence among such near-duplicate effects.
For example, Goldhaber and Brewer (2000) studied various types of teacher certification, and presented two models for each outcome. One model had 12 predictors, four of which represented different kinds of certification. The second model added interaction terms. If we are interested in the impact of, say, emergency certification, which model should provide the effect? Some meta-analysts would simply enter the slopes for emergency certification from both models (here, values for the math outcome of 0.58 and 0.27) and proceed with typical univariate analyses. However, such multiple effects from competing models are correlated, leading to violations of the independence assumption.
Becker et al. use large-sample theory to derive the correlations among slopes from regression models that share predictors, but vary in exact model composition. Let X1, X2, X3 and Y be standardized variables. From this set of variables four models can be produced that include X1; the slopes are b1, b1.2, b1.3 and b1.23, where bi.j represents the effect of Xi, controlling for Xj. Correlations were computed among these slopes for X1 as a predictor of Y. A grid of positive values for the correlations among Y and the Xs was explored; combinations leading to very small or negative matrix determinants were eliminated.
When all X-Y  values are equal, the correlations among slopes are quadratic functions, and decrease as values increase. Slopes are most highly correlated, with values approaching 1.0, when the other predictors are independent of X1. This makes sense, as the X1 slopes will be very similar across models if X1 is independent of the other Xs. Correlations among the slopes are also a function of the other X-Y correlations. For instance, the correlations between b1 and b1.23, and between b1.2 and b1.23 are quadratic functions of rY1, and the values on those curves are higher for higher values of rY3. Of greatest concern is that for small to moderate X-Y correlations, the correlations among the X1 slopes are high to extremely high, ranging upwards of .80 and even .90. In such cases meta-analysts essentially enter the same results twice, violating independence in the extreme, which impacts tests and measures of heterogeneity as well as standard errors (Jackson et al., 2012; Thompson & Becker, 2013). Recommendations are given for ways meta-analysts may proceed if studies report competing models.

Authors

• Betsy J. Becker, Florida State University

• Ariel M. Aloe, University of Iowa

• Ingram Olkin, Stanford University