Poster #27 - An Evaluation of the Alignment Method for Testing Measurement Invariance With Secondary Data and Complex Sampling Designs

• In Event: 3-001 - Poster Session 2 during Breakfast

Abstract

Background

Comparisons across groups/time are ubiquitous in developmental science. Underlying such comparisons is the assumption of measurement invariance (MI)—the same construct is being measured in the same way across groups/time (Kim & Yoon, 2011). Multiple-group confirmatory factor analysis (MG-CFA) is commonly used to evaluate MI (Brown, 2006). Unfortunately, standard stepwise procedures are error prone and unwieldy in the presence of many groups/items (Marsh et al., 2016). The newly developed alignment method (Asparouhov & Muthén, 2014), which involves fitting the unconstrained model and minimizing a loss function to align items and allow for group comparisons, offers a promising alternative. Yet, no simulations have evaluated this method when the sampling design necessitates weights. This is a critical gap due to (a) the prevalence of MI analyses involving secondary datasets generated from complex sampling designs (e.g., Niehaus & Adelson, 2013), and (b) research demonstrating that the performance of statistical estimators can suffer when weights are applied (Author). Thus, this simulation evaluated the alignment method across designs in which weights were either necessary or unnecessary.

Method and Results

Empirical Example
To maximize generalizability, the simulation parameters, population characteristics, and sampling design were based on empirical data drawn from the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011; NCES, n.d.) public-access dataset. The alignment method was applied to investigate MI of the Language/Literacy subscale of the Academic Rating Scale (NCES, 2002) across kindergarten children with different home languages (English, Spanish, Other). As shown in Table 1, five items exhibited non-invariance. Following alignment, the English group had greater teacher-reported language/literacy skills.

Simulation
The primary study factor was informativeness of the sampling design (weights necessary or unnecessary). Other factors included group size and degree of non-invariance. Preliminary results based on a subset of conditions and 100 replications indicated mostly adequate performance. Correlations between estimated and true factor scores were not significantly different between unweighted (r = .981) and weighted (r = .977) approaches. Table 2 provides relative bias of the estimators. Unweighted estimators were unbiased, whereas the weighted intercept estimators were positively biased (RB = .07).

Discussion

Secondary datasets provide rich information for conducting developmental research, including investigating MI across groups/time. Notably, these datasets stem from complex designs that necessitate sampling weights and alternative variance estimators. Most modeling techniques were developed outside of these contexts and thus their performance must be critically examined—the validity of our conclusions depends on the use of appropriate methods.

Author

• Natalie A Koziol, Nebraska Center for Research on Children, Youth, Families & Schools
  Presenting Author