- Browse
Search
On-Site Program Calendar
Browse By Day
Browse By Time
Browse By Person
Browse By Room
Browse By Unit
Browse By Session Type
- Information Menu
Search Tips
- Navigation and Settings Menu
Change Preferences / Time Zone
Sign In
- Social Media Menu
Facebook
Bluesky
Threads
X (Twitter)
LinkedIn
Instagram
YouTube
LLM-Driven Innovations in Psychometric Evaluation and Modeling
Fri, April 10, 1:45 to 3:15pm PDT (1:45 to 3:15pm PDT), InterContinental Los Angeles Downtown, Floor: 5th Floor, Wilshire Grand Ballroom ISession Type: Symposium
Abstract
This session presents five innovative applications of large language models (LLMs) in psychometrics. First, a novel method for generating interview-informed LLM synthetic responses is proposed to bridge qualitative and quantitative insights Second, various AI-generated persona methods are compared in terms of their variability and psychometric properties. Third, text-based approaches are explored to classify item difficulty levels for constructing parallel field-test forms. Fourth, the automation of Q-matrix construction is explored through the integration of LLMs and DCM results. Fifth, LLMs are used to quantify incidental content similarity to support computerized adaptive testing. A discussant will synthesize the strengths, limitations, and future directions. Collectively, these studies demonstrate LLMs’ potential to transform psychometric research through advanced modeling and automation.
Sub Unit
Chair
Papers
Leveraging Interview-Informed LLMs to Model Survey Responses: Comparative Insights from AI‑Generated and Human Data - Jihong Zhang, University of Arkansas at Fayetteville; Xinya Liang, University of Arkansas; Nicole Grace Bonge, University of Arkansas at Fayetteville; Anqi Deng, University of Arkansas at Fayetteville; Nicole Zarrett, University of South Carolina; Lin Tan, Virginia Tech
Variability in AI-Generated Personas: Method Comparison and Psychometric Assessment - Xinya Liang, University of Arkansas; Jihong Zhang, University of Arkansas at Fayetteville; Chunhua Cao, University of Alabama; Anqi Deng, University of Arkansas at Fayetteville; Nicole Zarrett, University of South Carolina
Leveraging LLM Tool Calling for Automating Q-Matrix Construction in Cognitive Diagnosis Analysis - Jihong Zhang, University of Arkansas at Fayetteville
Text-Based Approaches to Classification of Item Difficulty Levels - Hong Jiao, University of Maryland; Sydney Peters, University of Maryland; Hanna Choi, Ewha Womans University; Qingshu XU, Hunan Normal University
LLM-Enhanced Automated Item Content Analysis Supporting CAT Applications - Jing Huang, Purdue University; Jihong Zhang, University of Arkansas at Fayetteville; Hua-Hua Chang, Purdue University