Optimizing Cognitive Diagnosis CAT: Addressing Practical Challenges through Psychometric Innovation

• In Event: Toward a National Infrastructure for Equitable, AI-Enhanced Learning Diagnostics in STEM Education

Abstract

Introductory STEM courses are often known as “gateway” classes in college—courses like physics, chemistry, and math that open doors to future majors and careers, yet they are consistently associated with high drop, fail, and withdrawal (DFW) rates. Such facts raise an urgent question: How can we identify and support students who are falling behind—before it’s too late? To address this challenge, this project has developed a flexible and scalable Cognitive Diagnosis Computerized Adaptive Testing (CD-CAT) system, aims to pinpoint exactly what each student understands and where they are struggling early on. This system designed for high-enrollment STEM instruction and deployed it on the Learning About STEM Student Outcomes (LASSO) platform.
Working with data from over 20,000 students who took physics assessments on the LASSO platform, we identified several practical challenges in operationalizing CD-CAT at scale. To address the practical challenges, we refined key stages of the adaptive testing process—including test initialization and item selection—ensuring alignment with both psychometric goals and instructional needs. For instance, our algorithm adopted the Maximum Priority Index (MPI) method to guide item selection under strict constraints, balancing measurement precision with content requirements. This enhanced CD-CAT algorithm also extends beyond traditional attribute profile estimation by incorporating continuous ability measures, allowing for a more nuanced and personalized diagnosis of learning status. This dual-layer diagnostic model supports real-time, targeted feedback, even in large lecture settings.
To keep the assessment system growing and improving, we introduced a novel item online calibration procedure. This method allows new items to be calibrated during live testing without compromising the validity or fairness of the assessment. This is particularly critical in large-scale adaptive systems where item pools must evolve continuously to remain effective. We also tackled the issue of item parameter drift—a common but often overlooked challenge in adaptive testing, where repeated use of items may alter items’ difficulty or other characteristics. Our detection approach enables ongoing monitoring and adjustment, helping preserve fairness and comparability across test administrations. Using pre- and post-test data from 14,495 examinees on the LASSO platform, we applied this method to the Force Concept Inventory (FCI), a 30-item diagnostic test measuring two core attributes.
Beyond these technical advances, our work engages with complex instructional realities, such as hierarchical skills and multi-dimensional learning objectives, which are especially relevant in STEM fields. By accommodating these structures, the system can better model how students acquire and apply knowledge over time.
In sum, this project demonstrates how psychometric innovation can meet the practical demands of classroom assessment. By integrating cognitive diagnosis models, adaptive algorithms, and real-world constraints, our CD-CAT system offers a powerful tool for supporting learning, guiding instruction, and ultimately improving student outcomes in foundational STEM courses.

Authors

• Jing Huang, Purdue University

• Yuxiao Zhang, Purdue University

• Xiyu Wang, Purdue University

• Hua-Hua Chang, Purdue University