Approximate Measurement Invariance Methods for Harmonizing Classroom Observation Scores Across Disparate Conditions

• In Event: The Affordances and Limitations of Different Analytic Models for Data From Classroom Observation Systems

Abstract

Valid and reliable measurement of teaching is essential to evaluating and improving teacher effectiveness and advancing large-scale policy-relevant research in education. Despite the potential of classroom observations to anchor teaching quality assessments in specific and observable criteria, data produced by classroom observation systems tend to be heavily influenced by features of the observational environment other than the quality of teaching. One critical source of construct-irrelevant variation in classroom observations that has been largely neglected is item side variation. Conceptually, construct-irrelevant variation can be split to two principal sources—person side variation and item side variation. Person side construct-irrelevant variation arises when teaching quality varies across conditions (e.g., observations, lessons). Item side construct-irrelevant variation arises when the relationships between items and teaching quality (i.e., item parameters) vary across conditions. When item side construct-irrelevant variation is present, items function or reflect teaching quality differently across conditions and introduce measurement non-invariance such that assessments produced under different conditions are on incommensurate scales that do not preserve a common meaning and basis for comparison.

The goal of this study was to develop a set of psychometric methods that accommodates item and person side construct-irrelevant variation across lessons in order to separate teachers' persistent teaching quality from construct-irrelevant variance. The proposed methods introduce person and item side latent variables to relax assumptions of measurement invariance across conditions. With these methods, we can test, calibrate, and account for measurement non-invariance across multiple facets, assess correlates of non-invariance through structural components, accommodate high-dimensional n-level structures, and plausibly predict persistent levels of teaching quality free of construct-irrelevant variance. The proposed methods address critical gaps in the measurement of teaching because they conceptually model and bridge disparate measurement conditions through a tractable nonlinear mixed effect framework. The results of several classroom observation and student perception survey case studies suggest that non-invariance across conditions is the norm and that establishing an approximately invariant scale through the proposed methods improves model fit and the predictive validity of scores.

Authors

• Benjamin Kelcey, University of Cincinnati

• Kyle T. Cox, University of North Carolina - Charlotte

• Fangxing Bai, University of Cincinnati