Detecting Rater Bias in Mixed-Format Assessments

• In Event: Rasch Measurement Innovations

Abstract

Mixed-format assessments that include multiple-choice (MC) items and constructed response (CR) items that are scored using rater judgments include unique psychometric considerations. When these item types are combined, information about the psychometric quality of each component depends on that of the other. For example, the presence of differential item functioning (DIF) in MC items could compromise the sensitivity of rater effect indicators, such as differential rater functioning (DRF) indices. Likewise, the presence of DRF could compromise the sensitivity of DIF indices. We used real data and a simulation study to consider the impact of DIF on DRF indices, and the impact of DRF on DIF indices. Results indicate some interaction between DIF and DRF that varies across data collection designs.

Authors

• Stefanie A. Wind, University of Alabama
  Presenting Author

• Yuan Ge, The College Board
  Presenting Author