Unresolved Issues With Hu and Bentler Cutoffs and a Method to Derive More Appropriate Cutoffs

• In Event: Past, Present, and Future of Model Fit: Modern Solutions to Historical Challenges

Abstract

A central component of fitting a structural equation model is evaluating the fit of the model. To judge fit, cutoffs derived in the simulation study by Hu and Bentler (1999; HB) reign supreme, having amassed a quantity of citations that rivals articles containing science’s biggest breakthroughs. Despite the popularity of these cutoffs, they are known to have generalizability issues (Marsh et al., 2004). For instance, the cutoffs tend to under-reject misspecified models with many items (Kenny & McCoach, 2003) but over-reject acceptable models that have strong standardized loadings (Hancock & Mueller, 2011).

HB’s approach is essentially a power analysis. In a traditional power analyses, the goal is to determine the sample size at which statistical tests are sensitive to an effect. Similarly, HB’s goal was to uncover fit index values that were sensitive to a misspecification. However, just as sufficient sample sizes in power analyses are affected by model characteristics, so too are fit index values. This can be remedied by computing custom cutoff values for each specific model, as suggested by Millsap (2007). Unfortunately, this approach has not been embraced by empirical researchers, likely due to the high technical skill needed to execute an SEM simulation. Essentially, SEM is hard enough for empirical researchers without requiring that they also be experts in Monte Carlo methods to evaluate the fit of their models.

These difficulties have motivated us to develop a way to standardize and generalize this type of process so that researchers do not need to be fluent in Monte Carlo methods to properly evaluate the fit of their models. We term this method Dynamic Fit Indices and have created an algorithm that (a) creates a simulation design from a researcher’s model estimates, (b) executes the simulation study, and (c) derives a cutoff using a similar approach as HB but tailored to the characteristics of the model being evaluated.

Briefly, we reverse engineer a hypothetical model that would render the empirical model misspecified to a similar degree as in Hu and Bentler (1999) in order to determine what the distribution of fit index values would look like for a misspecified model in the subspace occupied by the empirical model. The advantage of this approach is that it does not require knowledge of the true model to capture the magnitude of misspecification. We establish this by replicating the model in HB’s seminal paper to demonstrate that our approach returns the same cutoff values and show how it can be generalized across all model subspaces.

In this presentation, we discuss how fit index cutoffs should more closely resemble power analyses such that cutoffs are recalculated for each model to ensure that the fit index cutoffs are actually sensitive to detect misspecification for characteristics of the model being evaluated. We review the algorithm that can replicate HB’s misspecifications and demonstrate its effectiveness. To make this approach accessible to empirical researchers, we also developed a Shiny application; a demonstration and tutorial of this software is provided in another presentation in this symposium proposal.

Authors

• Daniel McNeish, Arizona State University

• Melissa Gordon Wolf, University of California - Santa Barbara