Construct or Item Wording Effect? Evidence From a Multidimensional Item Factor Analysis of the Grit-S

• In Event: 61.013 - Co-Evolution of Substantive and Methodological Research: The Case of Grit, Self-Regulation, and Motivation

Abstract

Objectives
Duckworth and Quinn (2009) developed the 8-item Grit-S scale to measure grit and its two proposed components, perseverance of effort (PE) and consistency of interests (CI). However, the four items measuring CI are worded negatively while the four measuring PE are positively worded. This study addresses whether the latent variables CI and PE found in studies of grit’s factor structure reflect the existence of the constructs or the direction of the item wording.

Theory/framework
Many survey developers include negatively worded items to avoid respondents’ careless responses and re-focus their attention (e.g., Schriesheim & Hill, 1981). However, Woods (2006) and others found that including such items increases the chance of rejecting the null hypothesis of a unidimensional model even when the data generating the model is unidimensional. To address possible item wording effects some researchers have adopted item bifactor models to capture the item wording effect as residual dependency (Wang et al., 2018). However, because the Grit-S totally confounds construct (CI and PE) with wording type (negative and positive), researchers cannot disentangle the contributions of each to the common variance among items. Thus despite numerous studies examining grit’s factor structure (e.g., Duckworth & Quinn, 2009; Wolters & Hussain, 2015), it still is unclear whether the often-observed two factor simple structure of the Grit-S is due to the existence of the theorized sub-constructs or is an artifact of the positive and negative item wordings.

Methods
To decompose the item communalities into construct and item wording effects, we added eight items to the Grit-S; the new items had the same content as the Grit-S items, differing only in the direction of the wording. 561 Korean adolescents and 528 Korean college students completed the 16 items. We analyzed the data using three multidimensional item factor models (Figure 1) that include two two-tier item factor models (Cai, 2010) and a multitrait-multimethod model (e.g., Marsh & Grayson, 1995) to examine which model best represents the data, as well as to calculate the relative contribution of the construct or method (wording) factor to explain the variance of each item (communalities).

Results
Half of the item pairs did not exhibit significant item content effects once either methods or construct effects are accounted for in the two two-tier item factor models. The multitrait-multimethod model fit best for both samples, indicating both methods and construct effects should be modelled. The construct factors explain less than 50% of the common variance for six out of the eight original Grit-S items in the college sample and four out of eight in the adolescent sample. Overall and particularly in the college sample, there are substantial item wording effects that threaten the construct validity of two factor Grit-S structure.

Significance
The substantive significance of these results is that the often-reported two factor structure model of grit is due in large part to the directionality of the item wording rather than reflecting the constructs (especially for college students). Methodologically the study’s results have significant implications for survey item writing and the scale validation process.

Authors

• Hyo Jin Lim, Seoul National University of Education

• Ji Seung Yang, University of Maryland