Testing the Mediated Effect in Cross-Classified Multilevel Data

• In Event: Cross-Classified and Multiple Membership Random Effects Models: Demonstrations, Evaluations, and Extensions

Abstract

This study proposes an approach for testing mediation effects in cross-classified multilevel data where the initial treatment is assigned at the level of one crossed factor, the mediator is measured at the level of the other crossed factor, and the outcome is a level-1 outcome (thus, a 2->2->1 design). For example, a neighborhood characteristic might be hypothesized to influence a student outcome through a school characteristic with students cross-classified by neighborhood and school. Multiple membership and cross-classified random effects models are used to estimate the resulting indirect effect. We demonstrated the proposed model using ECLS-K data. In addition, we conducted a simulation study to assess the performance of the models in terms of the biases in the parameter estimates and the validity of the statistical inferences.

Method

Real Data Analysis

In the empirical data analyses, we extracted student, school, and neighborhood data in the spring of 1999 from the Early Childhood Longitudinal Study– Kindergarten Cohort study (1998). The indirect effect of neighborhood high-status residents on students’ math IRT scores via school academic atmosphere was examined. The sample data had a cross-classified multilevel structure. Students were cross-classified by schools and neighborhoods (defined by zip code tabulation areas). After deleting missing data and schools with fewer than 20 students, the final analysis sample consisted of 2,771 students, cross-classified by 130 schools and 128 neighborhoods.

Simulation Study

In the simulation study, we generated data mimicking the real data we analyzed. Two design factors were manipulated including the sample size and the degree of cross-classification. Following general recommendations about sample sizes for multilevel models, we considered two sample size conditions: (1) 500 students cross-classified by 50 schools and 50 neighborhoods, and (2) 2,000 students cross-classified by 100 schools and 100 neighborhoods. For the degree of cross-classification, we considered two conditions: fully cross-classified vs. partially cross-classified. In the fully cross-classified condition, students in a neighborhood had equal probabilities of being assigned to any school and vice-versa. In the partially cross-classified condition, 20% of the neighborhoods were selected and the students in each selected neighborhood were randomly assigned to only one of 2 schools. For the remaining neighborhoods, there was a one-to-one match between schools and neighborhoods.

Combining the two design factors, four conditions in total were examined. For each condition, 1,000 data sets were generated. Based on the generated data, the indirect effect was computed using the proposed method and the confidence interval were calculated using both the analytical and Monte Carlo approaches.

Results and Discussion

Simulation results showed that the proposed method produced consistent estimate of the indirect effect and reliable statistical inferences. However, the analytic confidence intervals tended to have less power than the Monte Carlo confidence interval estimates when sample sizes were relatively small. Results from the real data analysis and the simulation study will be included in the final paper. A fuller discussion justifying the need for this study and the reason for the simulation conditions will be provided as well as directions for future research.

Author

• Wen Luo, Texas A&M University - College Station