Navigating Challenges in Operational Psychometric Environments

Session Type: Coordinated Paper Session

Abstract

Operational psychometric work presents unique challenges due to the intricate demands for accuracy and fairness across diverse assessments. This coordinated session convenes experts, practitioners, and researchers from various organizations to collectively address these pressing challenges. Our goal is to identify and share practical strategies and solutions that can be expediently deployed to mitigate these challenges.
The first paper leverages natural language processing techniques to predict the response time in adaptive Next Generation Science Standards tests and proposes to integrate it into the item development process. Subsequently, the second and third papers tackle challenges of local independence violations in item response theory. Specifically, they address two distinct situations: discrepancies in scoring rubrics within writing tests, and uncontrolled speed at the item-level in timed adaptive exams. Both papers aim to achieve precise item and ability estimates, offering unique solutions for each scenario. The last two papers address the complexities of collaborating with subject matter experts, such as careful management between psychometricians and subject matter subjects, and test construction review, with the objective of optimizing the development of assessment by merging the strengths of both domains.

Sub Unit

• Individual Papers, Innovation Demonstrations, Coordinated Sessions, or Organized Discussions

Session Organizer

• Yawei Shen

Chair

• Siyu Wan, ABIM

Individual Presentations

• Predicting Response Times for NGSS-Aligned Item Development - Mina Lee, Cambium Assessment; Wanchen Chang, Cambium Assessment; Wei Schneider, Cambium Assessment; Zebing Wu, Cambium Assessment; Suhwa Han, Cambium Assessment, Inc.

• Adjusting Item Parameter Inflation Caused by Local Item Dependence in Writing Items - Hongyu Diao, American Board of Internal Medicine; Larissa Smith

• The Effects of Uncontrolled Speededness on IRT Item and Ability Estimation - Jaime Malatesta, GMAC; Paulius Satkus, GMAC; Kyung (Chris) T. Han, GMAC

• Collaborating with Subject Matter Experts - Janet Mee, NBME

• Best Practice for Review Test Form Constructed from Content Experts - Yang Lu, Pearson; Yawei Shen; Mustafa Kuzey Bilir, PEARSON

Discussant

• Jiawei Xiong, CURRICULUM ASSOCIATES