Comparing Reliability Estimation Methods in Cognitive Diagnosis Models

• In Event: Marriott Roundtable Session 70
  In Roundtable Session: Cognition and Assessment SIG Roundtable 2

Abstract

Practical applications of CDM remain nominal relative to other psychometric methods despite growing research and policy interest. In some part, this is related to the limited research on key aspects of test interpretation and utilization, including reliability. This paper describes four extant methods developed for estimating reliability in the CDM framework. Interestingly, none of the papers presenting these methods acknowledge or build upon one another’s work. This paper seeks to synthesize and compare these methods. First, we conceptualize the methods in terms of classical test theory and compare the methods. After this discussion, the paper proposes to conduct analysis of simulated and actual test data to compare the reliability coefficients estimated from each method.

Authors

• Justin Paulsen, Indiana University - Bloomington

• Shenghai Dai, Washington State University - Pullman

• Xiaolin Wang, The University of Kansas