The Impact of Ignoring Multiple Membership in Three-Level Multiple Membership Random Effects Modeling

• In Event: Cross-Classified and Multiple Membership Random Effects Models: Demonstrations, Evaluations, and Extensions

Abstract

Objectives

This study investigates the impact of ignoring multiple-membership at either the top or the intermediate level of a three-level multiple-membership random effects model (MMREM). For multiple-membership multilevel data, MMREMs have been used to investigate the relationships among variables within a given level and across levels (e.g., Goldstein, 2010; Rasbash, & Browne, 2008). A simulation study by Chung and Beretvas (2012) compared MMREM to a mis-specified traditional multilevel model that ignored two-level multiple-membership data structures. However, this study only provided partial results on the impact of ignoring multiple-membership data structures because it examined only two-level multiple-membership data structures. In fact, three-level data are fairly common in social and behavioral science fields. Moerbeek (2004) investigated the impact of mis-specifying a level of nesting in a conventional three-level multilevel model and revealed that the biases depended on which level (i.e., either the intermediate level or the top level) was ignored. Luo and Kwok (2009) examined the effect of ignoring a crossed factor in a three-level cross-classified random effects model and reported that the magnitude and sign of the observed biases depended on the level of the ignored crossed factor. Thus, further investigation is needed to provide a more complete picture of the impact of mis-specifying MMREMs in a three-level modeling framework.

Methods

The current study conducted two main investigations to examine the impact of mis-specifying MMREMs for a three-level multiple-membership data structure. Study 1 investigated MMREMs in which the multiple-membership occurred at the intermediate level. Study 2 investigated models with multiple-membership at the top level. Data were generated according to a three-level MMREM with 20% of participants experiencing multiple-membership at either level-two or level-three. The generating values for fixed effects were 1 for the intercept, and 0.4 for the predictor at each level of the model. The residuals were sampled from normal distributions with means of zero and variance of 0.5 for level-one, 0.2 for level-two, and 0.2 for level-three. The MLwiN software (version 2.32) was used to generate 1,000 data sets and estimate two models: the traditional three-level model and the correct three-level MMREM.

Results and Discussion

As shown in Table 13, study 1 (i.e., ignoring the multiple-membership at the intermediate level) resulted in under-estimated variance components at the intermediate level, over-estimated variance components at level 3, and substantially over-estimated variance components at level-1, while also leading to underestimation of level-two predictors’ coefficients (i.e., at the level at which the multiple-membership occurred). The final paper will provide fuller details about the results for study 1 as well as including results for study 2. A fuller justification supporting the need for this set of studies and for the conditions that were examined will be provided along with implications and guidelines for applied researchers.

Authors

• Hyeonjeong Jeon, Chungnam National University

• Hyewon Chung, Chungnam National University