Evaluation of Q-Matrix Validation Methods Under Attribute Hierarchies

• In Event: Advances in Psychometric and Machine Learning Approaches to Assessment

Abstract

Various Q-matrix validation methods have been developed to ensure the validity of inference for cognitive diagnosis. Their performance in scenarios involving hierarchical attribute structures has not yet been investigated. This study compares four Q-matrix validation methods (GDI, Stepwise, Hull, and MLR-B) across four attribute hierarchies (linear, convergent, divergent, mixed). Findings indicate that the Hull method is best for minimizing overspecification by controlling for unnecessary attributes. The GDI or Stepwise methods better prevent the omission of critical attributes, though they may increase overspecification. For balanced performance in accuracy, complexity, and efficiency, the MLR-B method is the most suitable choice. These results highlight the significant impact of attribute hierarchies on Q-matrix validation and provide guidance for developers.

Authors

• Tianpeng ZHENG, Peking University

• Zhehan Jiang, Peking University