Cross-Classified Meta-Analytic Models

• In Event: Cross-Classified and Multiple Membership Random Effects Models: Demonstrations, Evaluations, and Extensions

Abstract

In a meta-analysis, we combine the data from multiple studies in order to obtain more accurate estimates of the parameters of interest, to increase the statistical power in testing these parameters, and to study the possible moderating effects of study and sample characteristics. In a typical meta-analysis data have a hierarchical structure: study participants are nested in studies. A meta-analysis can therefore also be regarded as a multilevel analysis, and estimation algorithms and software developed for multilevel models are indeed commonly used for executing meta-analyses.

The multilevel framework offers an elegant way to deal with dependent effect sizes. A three-level model with an additional upper level can be used for instance when studies can be grouped in research groups, accounting for the relative resemblance of the results from studies from the same research group. A three-level model with an intermediate level can be used to account for dependencies within studies, for instance when within-studies effect sizes are calculated for multiple outcomes or for different subpopulations.

In this paper, we propose a cross-classified model to model the dependence between effect sizes within studies resulting from the measurement of multiple outcomes and multiple subpopulations at the same time. The model includes random effects for outcomes and for subpopulations. Outcomes and subpopulations are crossed factors: an outcome can be measured in multiple subpopulations, and in a subpopulation multiple outcomes can be measured.

Method

The cross-classified meta-analytic model is illustrated using data from a real meta-analysis example. In addition, a simulation study has been performed that is intended to assess the performance of the use of this model as compared use of three-level models in which one of the crossed factors is ignored. We manipulate a number of design conditions and generating parameter values to examine their impact on parameter recovery.

Results and Discussion

The results of the simulation study support that the cross-classified model is to be preferred if there is dependence caused by multiple factors, whereas the nested three-level models are preferred when there is dependence because of a single factor. We conclude that careful consideration of the underlying data structure is required in order to guarantee appropriate estimates. In the final paper we will have room to provide detailed information about the model’s parameterization, justification for scenarios in which the cross-classified model might prove useful for meta-analysis and explanation of how to interpret the model’s parameters. In addition, we provide far more detail about the simulation study that has already been completed including rationalization for the conditions that were examined and a full discussion of the results and associated implications for applied meta-analysts. We also provide directions for future methodological research.

Authors

• Belen Fernandez-Castilla, Katholieke Universiteit Leuven

• Eline Belmans, KU Leuven

• Wim Van den Noortgate, Katholieke Universiteit Leuven