Moving Beyond Hu and Bentler: Comparing Adjusted Cutoffs From Dynamic Fit Indices and Equivalence Testing

• In Event: Past, Present, and Future of Model Fit: Modern Solutions to Historical Challenges

Abstract

Fit index cutoffs suggested by Hu and Bentler (1999) dominate model fit evaluation in empirical studies. Since 1999, many studies note that these cutoffs do not generalize across model characteristics including the number of items (Kenny & McCoach, 2003; Shi et al., 2019); degrees of freedom (Chen et al., 2008; Kenny et al., 2015); or standardized loading magnitude (Browne et al., 2002; Hancock & Mueller, 2011). Recent methods have been suggested to adjust fit index cutoffs depending on model characteristics, specifically the equivalence testing (EQT) approach of Yuan et al. (2016) and the dynamic fit index (DFI) approach of McNeish & Wolf (2021).

The EQT approach allows researchers to test whether a model contains a prespecified magnitude of misspecification to draw conclusions about the quality of the model. Using traditional thresholds for different magnitudes of misfit are a logical choice for the prespecified value required for EQT. However, Yuan et al. (2016) note the issues with cutoffs and instead propose an adjustment. Yuan et al. (2016) simulate χ2 test statistics from a variety of sample sizes and degrees of freedom and fit a best subset regression to the simulation results. The coefficients from this empirical best subset regression model are used to correct the cutoff bins originally suggested by MacCaullum et al. (1996) designed to demarcate “Excellent”, “Close”, “Fair”, and “Mediocre” fit.

We compare this EQT approach to the three levels of misspecification tested in the DFI approach discussed in other presentations in this symposium. The main goal is to compare the performance of these different approaches to adjusting fit index cut-offs to determine which is best suited to reject misspecified models and retain correct models. We will generate data from two different models (one with two-factors and one with three-factors). We then will fit correct and misspecified models to each generated dataset. The misspecified models with be either have the incorrect factor structure or misspecified relations between the variables. Manipulated conditions are based on the characteristics noted to affect fit index values: sample size (250 or 1000), items per factor (4, 8, 12), and standardized loadings (0.45, 0.60, 0.75, and 0.90).

To present some preliminary results, we performed 100 replications for the 12-item, N=1000 condition for a two-factor generating model that fits a misspecified one factor model. Figure 1 shows the data generation model (top panel) and the fitted model (bottom panel). Table 1 shows the percentage of replications classified at different levels of misfit severity using RMSEA of CFI with cutoffs produced by DFI and EQT. Notably, the DFI cutoffs consistently classify misfit across conditions and indices. However, sensitivity of CFI with EQT is heavily dependent on the standardized loading condition. This may suggest that EQT may not adequately account for all relevant characteristics that affect cutoffs because it only includes sample size and degrees of freedom in its adjustment. The custom simulation approach of DFI does not assume a functional form and performed more consistently. The full simulation will assess whether this pattern is upheld more generally.

Author

• Daniel McNeish, Arizona State University