Automated Generation of Verbal Analogical Reasoning Items: An AI-augmented Cognitive Item Design Approach

• In Event: AERA Roundtable Session Saturday 9:45 am - Gold 4
  In Roundtable Session: AI in Assessment and Reasoning: Innovations in Measurement and Scale Development

Abstract

Automatic item generation (AIG) enables rapid test development, yet current AIG methods often prioritize efficiency over validity. This study pioneers a novel hybrid AIG: the AI-augmented cognitive item design approach, integrating AI and cognitive item design for the first time in automated item generation. Applied successfully to verbal analogical reasoning items, the framework involves: (1) cognitive modeling to identify stimulus features affecting processing stages, (2) operationalizing features via GPT-4o for template development, and (3) mass item production and evaluation. This study expands the methodological horizons of AIG, and seamlessly integrates cognitive psychology with psychometrics, optimizing both efficiency and validity in item generation.
Key words: Automatic Item Generation; AI; Cognitive Item Design Approach; Verbal Analogical Reasoning Items

Authors

• Yue Wang, East China Normal University

• Xiangdong Yang, East China Normal University