A Framework for Examining Structural Aspects of Construct Validation in Multilevel Settings

• In Event: Problems With Interpretations of Multilevel Data: Extending Research Beyond Hierarchical Linear Modeling

Abstract

Confirmatory factor analysis (CFA) is ubiquitous when validating psychological measures. A model is evaluated: X=λξ+ ε, where X is an n x p matrix of observed responses on a set of p items, λ is a p x k matrix of item loadings, k is the number of hypothesized constructs with covariance matrix ϕ and ε is an n x p matrix of error terms, with covariance matrix θ. Support for the hypothesized structure is gained if the model is found plausible given observed item covariances. Complications arise in the context of data nesting because structural validation can occur at within- and between-cluster levels; researchers suggest validating (Geldhof, Preacher, & Zyphur, in press), and conducting reliability analyses at both levels (Raykov, 2009; Raykov & Penev, 2001). Absent in these approaches is a discussion of whether a multilevel approach to validation is conceptually appropriate.
In this presentation a framework to address decisions faced in structural validation in multilevel contexts is provided. Within this framework, determining the future use of the measure(s) is addressed as it relates to how to conduct validation analyses. To illustrate, if, in the future, a measure will only be used within a single cluster (e.g., by classroom teachers) then the within-cluster portion should be of validation concern. However, if the measure will be used across a large geographic region with non-nested data (only one student is selected per cluster) then the within-cluster structure is not relevant and an aggregate validation process is appropriate. Additional components of the framework (e.g., validation of contextual versus climate constructs) will be demonstrated by using data from grade 8 students and their parents (N= 8,070 students in 1,144 schools) from the Early Childhood Longitudinal Study of Kindergarten (Tourangeau, Nord, Le, Sorongon, Najarian, & Hausken, 2009).
Preliminary results demonstrate how items can be used differentially to validate two types of constructs: climate and contextual (Marsh et al., 2012). A climate construct (school-level of supportiveness) was derived by combining six items: How often are the following true? a. My classmates think it is important to be my friend; they like me the way I am; they care about my feelings and like me as much as they like others. e. My classmates really care about me). A contextual construct (parental support) was derived by combining five items (as provided by parents of 8th grade students): In the past year, how often have you talked with {CHILD} about smoking and tobacco use, drinking alcoholic beverages, sex and drug use?. Steps to conduct each of the two validation approaches will be provided alongside a conceptual roadmap for how to conduct these analyses within an applied setting.
This study addresses an important problem in validation research, which is the absence of a conceptual and methodological framework for conducting multilevel validation in the cases when such a procedure is appropriate and guidance in discerning when it is not. Therefore, this study makes an important contribution to the field of measurement by filling in this gap in the literature.

Author

• Laura M. Stapleton, University of Maryland