Validity and reliability of Socioeconomic Status measurement in a regional ILSA using a causal-formative model.

• In Event: Critical perspectives on large-scale assessments: Methodological innovations, validity, and societal implications

Proposal

Over the past few decades, International Large-Scale Assessments (ILSAs) have gained popularity around the globe as a source of empirical data to understand differences in educational achievement across countries. Governments and policymakers expect ILSAs to provide valuable insights into their own educational systems, by comparing them with other countries (Rutkowski, et al., 2020). Moreover, the expanding technologies and support for assessments have provided new efforts and interest to countries in ILSAs, as they decrease the costs of data collection and processing (Liu & Steiner-Khamsi, 2022).
In addition to the achievement assessment, ILSAs commonly use questionnaires to gather information about context that could provide explanations for the variations perceived on the main survey measure (Schulz & Carstens, 2020; Hooper, 2022). These background questionnaires have become crucial to the validity of the estimation of achievement scores (Rutkowski & Rutkowski, 2010), and thus their trustworthiness to researchers and policymakers (Wagemaker, 2020).
Assuring questionnaire quality has become one major limitation for ILSAs (Rutkowski, 2018). In particular, families’ socioeconomic status (SES) has become a relevant source of information to contextualize educational achievement (Avvisati, 2020; Atteberry & McEachin, 2020), because it provides an understanding of the mechanisms by which family SES correlates to academic achievement and isolated influences of other contextual characteristics (Caro & Cortés, 2012).
One particular challenge of the SES indicator is its capacity to provide valid and reliable country-comparison information. Emerging debates are seeking to understand whether regional ILSAs targeting educational systems with greater homogeneity in language, culture, or geography, are better suited to overcome the challenges of comparability that global assessments face (Braun & Singer, 2019; Hernandez-Torrano & Courtney, 2021). In addition, regional ILSAs represent important cases of international data justice, by providing valid and reliable data that is context-adequate for non-western countries (Neuschmidt, et al., 2022; Howie, 2022).
In the case of ERCE, a regional ILSA within Latin America where participating countries mainly share the same language and have partly shared cultural roots, only recent and scarce evidence has been presented about its technical challenges (Gustafson & Barakat, 2023; Carrasco et al., 2023). Even more scarce are the technical studies assessing the comparability of the SES indicator across its participating countries (Sandoval et al., 2019).
However, the measurement of SES is still a paramount challenge: conceptually, there is no common definition on its construct, and practically, it has been shown to lack adequate psychometric properties for cross-cultural comparisons (Antonopolis, 2023; Sandoval et al., 2019; Rutkowski & Rutkowski, 2013; Eryilmaz, et al., 2020). A major debate has been held around the nature of the models that should be used to study SES: reflective models or causal-formative models (Antonopolis, 2023; Howell, et al., 2007). Although many influential psychometric analyses done on the SES indicator of ILSAs use reflective models (Sandoval et al., 2019; Caro et al., 2014; Rutkwoski & Rutwkoski, 2013; Rew et al., 2022), it has been argued that this could yield errors of model misspecification (Bollen & Bauldry, 2011; Bowen & Diamantopolous, 2017).
Strong arguments support that SES in the ILSA context does not comply with the assumptions of reflective models, namely because it lacks a unifying theoretical and internally consistent construct, the items used to measure it are not interchangeable, the directions of the causality go from the indicators to the constructs, and because the concept itself is completely defined by its indicators (Caro & Cortés, 2012; Diamantopolous, et al., 2008).
This paper provides evidence for assessing the validity and reliability of the SES indicator of ERCE as a regional ILSA while overcoming methodological challenges of SES measurement. While other studies have provided the comparability of the SES indicator across global and regional ILSAs (Sandoval, et al., 2019), they have relied on reflective models, thus being prone to measurement misspecification (Bollen & Bauldry, 2011; Bowen & Diamantopolous, 2017). This paper uses recent directions on the estimation of validity and reliability of causal-formative indicators (Caro & Cortés, 2012; Gruijters, Fleuren, & Peters, 2021; Ellwart & Konradt, 2011). Finally, the paper will present a discussion on the fairness of ILSAs questionnaire by comparing reliability across countries and its relationship with national income, following research that shows equity issues on the comparability across ILSAs (Rew et al., 2022; Gustafson & Barakat, 2023).
This paper analyses the families’ socioeconomic indicator (ISECF) of the Estudio Regional y Comparativo de Educación (ERCE) on its 2019 wave. The indicator is composed by 18 items asking about parental education and occupation, housing conditions, and home possessions (UNESCO, 2022). Data comes from questionnaire responses of parents and caregivers of 6th graders across 16 countries.
Following specialized guidelines, this study analyzes internal consistency using a set of estimators (Cronbach’s alpha, the lower greater bound -or beta-, and the greater lower bound), compares item weights across countries, and uses a benchmark variable to support the validity of the SES measurement (Caro & Cortés, 2012; Bollen & Lennox; 1991; Diamantopoulos, et al., 2008).
Findings show a variation in internal consistency estimates (ranging from .52 to .83 on the lowest greater bound) which is lower than those on PISA results (Caro & Cortés, 2012). In addition, a strong negative correlation (-0.31) is shown between the bounds and the national income of countries. Overall, similar results are found using other internal consistency bounds and correlations.
Results on validity and reliability of the SES measurement in ERCE 2019 seem to provide support for regional ILSAs being better suited for cross-country comparison, compared to benchmark analyses of PISA. However, differences in the SES item weights show that the interpretation of SES across countries in Latin America should be taken with caution.

Author

• Andres Fernandez-Vergara, University of California, Los Angeles