An Approach to Investigating Construct-Irrelevant Variance for Contextualized Constructed-Response Assessment

• In Event: Evaluating, Optimizing, and Monitoring Rater Performance

Abstract

Construct-irrelevant variance (CIV) is of significant importance for validity of contextualized constructed response assessment. Here, we develop a model including CIV, due to three factors: variability of scenarios, judging severity, and rater sensitivity to scenarios. We propose using a many-facet Rasch measurement to examine the CIV because this model can compare different CIV factors on a shared scale. Applying the model to a video-based teacher pedagogical content knowledge assessment, we find the CIV due to the variability of scenarios is substantial, while the CIV due to judging severity and rater sensitivity of scenarios are not, for assessing teachers’ identification of student thinking; For assessing teachers’ identification of responsive teaching, the CIV due to the three proposed factors are all substantial.

Authors

• Xiaoming Zhai, University of Georgia

• Kevin Haudek, Michigan State University

• Molly A.M. Stuhlsatz, BSCS Science Learning

• Christopher D. Wilson, Biological Sciences Curriculum Study