Modeling Latent Constructs for Multilevel Observational Data in Education: Multilevel Modeling (MLM) Versus Multilevel Structural Equation Modeling

• In Event: Composite/Factor Scores Analyses With Bias Correction: Good Alternatives to Full Structural Equation Modeling?

Abstract

Educational researchers frequently work with data measured as multilevel structures; sometimes, they are also interested in latent constructs that cannot be directly observed and measured. Therefore, handling data dependency and measurement error issues is particularly important in statistical modeling. Multilevel Structural Equation Modeling (MSEM) is a promising approach to dealing with both issues. However, educational researchers still prefer Multilevel Modeling (MLM) to MSEM. Conventional MLM cannot address the data dependency issue in within-level predictors. In addition, it cannot include a measurement model to handle measurement errors and construct a latent factor. As such, computing an average score to represent a latent factor in MLM is a common alternative approach in educational studies, especially for studies using the Trends in International Mathematics and Science Study (TIMSS) data.

The present study aimed to conduct a simulation study to evaluate the consequence of using an average score to represent a latent factor in MLM. Given that MSEM properly handles both data dependency and measurement error issues, the MLM simulation results are used for comparison with MSEM results. The discrepancy between them would be considered as a bias. The simulation factors included the level of the intraclass correlation coefficients (ICC) for the predictor and outcome, the level of factor loadings of the latent predictor at the between- and within-levels, as well as the multiple centering strategies. The simulation followed the multilevel settings of TIMSS (i.e., the cluster size of TIMSS is about 30, and the average number of clusters is approximately 150). The results will then provide guidance on selecting adequate modeling strategies under a variety of complex scenarios.

The simulation results indicated that when both data dependency and measurement error issues are involved in MLM, group-mean centering and latent-mean centering demonstrated the capacity to partition the variance into the between level and within level across various conditions. However, these two centering approaches showed the limitations of handling measurement errors. The findings suggested that the cutoff criterion of measurement errors for group-mean centering and latent-mean centering should be each within-level factor loading ≥ 0.8 (i.e., each measurement error variance < 0.36; within-level composite reliability ω ≥ 0.88). In some situations (e.g., within-level factor loading = 0.5), grand-mean centered MLM showed better accuracy (Root Mean Square Error) than the group-mean centered and latent-mean centered options. It was because measurement errors and data dependency luckily canceled out each other. Despite that, the grand-mean centered MLM yielded unacceptable relative parameter bias and relative standard error bias under the most conditions. Hence, grand-mean centered MLM (or uncentered MLM) is not recommended.

In sum, this study suggested that the bias of using an average score to represent a latent predictor in MLM is acceptable only when the following criteria are met: (1) group-mean centering or latent-mean centering is utilized; (2) the within-level factor loading of each item is equal to or above .80 (i.e., within-level composite reliability ω ≥ 0.88). Otherwise, MSEM is recommended.

Authors

• Chi-Ning Chang, The University of Kansas

• Oiman Kwok, Texas A&M University - College Station