Moderating Measurement Noninvariance in Predictive Equations

• In Event: Addressing Practical Issues in Measurement Invariance Research: Methodological Investigations and Advances

Abstract

Objectives
Measurement invariance is essential for the predictive validity of psychological constructs when making cross-cultural comparisons. The objective of this study is to use multigroup confirmatory factor analysis (MGCFA) to develop moderator variables based on the lack of invariance in an international dataset. Through regression analysis, improvement in prediction with the moderators is examined.
Perspectives or Theoretical Framework
Understanding noninvariance resulting from cultural differences is important for accurate cross-cultural comparisons (Church, 2010; Hancock, 1997). The use of noninvariant measures can compromise the predictive validity (Millsap, 2011) of cross-cultural models due to biased regression estimates. A measure of noninvariance that increases the predictive validity of scores may be employed to moderate the noninvariance and correct inaccurate prediction across cultures.
Measures from the Programme for International Student Assessment (PISA) survey (OECD, 2014) are known to lack invariance across countries (OECD, 2014). Thus, several PISA mathematics psychological constructs will be examined to create moderator variables capturing noninvariance. We asked the question: Can dichotomous grouping variables based on changes in model fit indices be useful moderators in a predictive equation?
To answer this question, first, we examine the invariance of each construct and produce three dichotomous measures to model noninvariance at the metric and scalar levels. Second, each construct and their respective invariance measures will be tested in regression equations predicting achievement. We expect the moderators to increase predictive accuracy.
Method
Sample
The 2012 PISA data were used (OECD, 2014) from 485,490 15-year-old students across 68 countries.
OECD Constructs
The PISA constructs for mathematics self-efficacy, anxiety, interest in math, and math work ethic (OECD, 2013; 2014) will be examined for noninvariance and serve as primary predictors in individual regression equations predicting mathematics achievement.
Procedure
MCFA models will be estimated using MPLUS 7.1 (Muthén & Muthén, 2012). Seven thousand randomly selected cases from the full sample will serve as the reference group for each model to avoid problems associated with large size disparities between groups (Chen, 2007).
The reference group is used to establish a 1-factor baseline model for each construct. Individual focal countries are then compared with the reference group using progressively constrained models (Marsh et al., 2006). Change thresholds in the fit indices (RMSEA, CFI, SRMR) between constraint levels determine noninvariance (Chen, 2007) and are used to calculate the moderator variables for metric and scalar invariance.


Results
The complete analysis and paper with the mathematics variables will be ready in February, 2016. Results on pilot data (NEO personality traits; McCrae, et al., 2005) support that predictive validity is improved with invariance moderators. Table 1 presents regression results illustrating a significant invariance moderator. Figure 1 illustrates the interaction of the invariance moderator with the predictor.
Scientific Significance / Practical or policy implications
The significant performance of invariance moderators may improve the predictive validity of domain scores and help researchers make accurate cross-cultural comparisons. This is especially helpful when using well-established large-scale data, such as PISA, produced by third-party organizations where the researcher has no control over the items or survey implementation.

Authors

• Brian F. French, Washington State University

• Bruce W Austin, Washington State University - Pullman