Dynamic Model Fit Indices: A R Shiny Application

• In Event: Past, Present, and Future of Model Fit: Modern Solutions to Historical Challenges

Abstract

In SEM, approximate goodness of fit indices (AGFI) are essentially effect size measures that can be used to evaluate if the model fit is still acceptable despite some misfit (Mulaik, 2007). Hu and Bentler (1999; HB) conducted a seminal simulation study to provide some guidance on the proper use of AGFI in practice, resulting in a set of fixed cutoff values that are often improperly applied beyond the simulation subspace (Marsh et al., 2004). HB’s simulation utilized one model: a three-factor model with 15 items, all with loadings ranging from .70 to .80, and varied three conditions: degree of misspecification, sample size, and normality. However, the magnitude of AGFIs are dependent upon the number of indicators and factor reliability. Specifically, models with higher loadings will appear to fit the data worse when compared against HB’s fixed cutoffs (McNeish et al., 2018).

Millsap (2007) suggested that HB’s cutoffs could be recalculated for each researcher’s model of interest to ensure that the fit index cutoffs are appropriately scaled according to the characteristics of a particular model. This practice has not been embraced by empirical researchers, likely because it requires expertise in Monte Carlo simulation, coding, and a-priori knowledge of the misspecifications of concern.

To remedy this, we introduce a set of R Shiny applications called Dynamic Fit Index (DFI) Cutoffs, which automates the process of generating DFI cutoffs that are tailored to the user’s specific model (Figure 1). These applications can be accessed at www.dynamicfit.app, and currently include two applications relevant to this presentation: a multi-factor CFA and a one-factor CFA. The applications only require two pieces of information from the user: their model statement with standardized loadings in a .txt file, and their sample size (Figure 2). After uploading these, the user will press submit and the application will generate a series of misspecified model statements and run a Monte Carlo simulation for each misspecification (each with 500 replications) to return the model-specific cutoffs.

Despite only recommending one fixed set of cutoffs, HB initially simulated two successive levels of misspecification (Table 1). To encourage researchers to properly conceptualize AGFI as effect size measures (as they were originally intended) and to account for model complexity, we extend HB’s approach to return a continuum of misspecification cutoffs. In multi-factor CFAs, this is done by consecutively omitting a total of j-1 cross-loadings from the true model (where j is the number of factors) and returning a cutoff value for each sequential level of misspecification severity. We extend this framework to one-factor models by successively omitting residual covariances from indicators, proportional to the total number of indicators in the model (Shi & Maydeu-Olivares, 2020; Table 2).

In this presentation, we will review the continuum of misspecification algorithm for both models, demonstrate a tutorial of the Dynamic Model Fit application, and discuss the implications for scale validation in education and psychology using SEM.

Authors

• Melissa Gordon Wolf, University of California - Santa Barbara

• Daniel McNeish, Arizona State University