The Impact of Cluster Size and Intraclass Correlation on Cross-Classified Multiple Membership Random Effects Models

• In Event: Cross-Classified and Multiple Membership Random Effects Models: Demonstrations, Evaluations, and Extensions

Abstract

In the social sciences, data are commonly collected with a complex underlying structure. Multiple membership data structures, for example, are commonly encountered in educational data sets in which student work or teacher performance is graded by more than one rater. Fortunately, extensions to the multilevel model allow us to fit models that accommodate this type of underlying complexity; cross-classified multiple membership random effect models (CCMMrems) have been developed to fit data structures that include observations belonging to more than one classification cluster (see e.g., Browne, Goldstein, & Rasbash, 2001).

When using these types of multilevel models, parameter estimation is complicated when the outcome variable of interest is binary, because intractable integrals must be evaluated requiring numerical or Monte Carlo integration techniques. When estimating more general random intercept logit models, for instance, the level-2 variance is prone to bias, particularly when the intraclass correlation (ICC) value is low and the number of level-2 units is small (Austin, 2010; Browne & Draper, 2006; Grilli, Metelli, & Rampichini, 2014; Rodrıguez & Goldman, 2001).

This bias in level-2 variance components is especially concerning when using CCMMrems, because the models have more than one level-2 variance component, and the level-2 random effects variance estimates are often of interest to researchers. Yet, the impact of cluster size and ICC on CCMMrem parameter estimates has not been studied. The purpose of this paper, therefore, is to evaluate the impact of cluster size and ICC on CCMMrem parameter estimation when the outcome variable of interest is binary.

Method

To evaluate that impact, we conducted a simulation study to examine the performance of CCMMrems with binary outcome variables. We mimicked an operational testing scenario where multiple raters were randomly assigned to grade examinees yet only one outcome score per examinee was retained in the final dataset. Ten dichotomous item scores were generated for two raters per examinee. When those two raters agreed, that consensus score was the final item score. When the raters’ scores disagreed, a third item score was generated. The modal score of the three ratings was then assigned as the final score.

We considered twelve balanced designs varying the number of clusters (i.e., raters), j = 10, 40, 70, 100 crossed with multiple membership participants (i.e., examinees), i = 10, 30, 50. Examinee abilities were sampled from a standard normal distribution, and rater effects were sampled from a normal distribution with a mean of zero and the following standard deviations: 0.5, 1, 1.5, and 2.5. The corresponding variances, precisions, and ICCs for those conditions are shown in Table 22. One thousand datasets were generated for each condition.

Estimates were produced using MCMC estimation as implemented in MLwiN (version 2.27). Preliminary results support previous research indicating that the level-2 variance estimates are prone to bias. The magnitude of bias diminishes as the value of the ICC becomes larger and the number of multiple membership units and members per unit increases. Final results and implications for practitioners and recommendations for future research are discussed.

Author

• Daniel L. Murphy, Pearson