Is AI Smart Enough to Identify the Right Factors? A Pilot Study on AI-Based Scale Development for Subjective Topics

• In Event: AERA Roundtable Session Saturday 9:45 am - Gold 4
  In Roundtable Session: AI in Assessment and Reasoning: Innovations in Measurement and Scale Development

Abstract

This study explores the use of generative artificial intelligence (AI) for attitude scale development in the domain of attitude measurement. Building on the Attitude Toward Censorship Scale (Thurstone, 1931), we employed ChatGPT-4 to generate new items through iterative prompt engineering and organized them into thematic categories. Data from 227 university students were analyzed using exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) to compare AI-suggested structures with empirically derived solutions. Results indicated that while the AI-generated five-factor structure demonstrated acceptable fit, more parsimonious two- or three-factor models aligned better with empirical data. Findings suggest that AI can broaden conceptual item spaces and enhance efficiency in scale development while requiring psychometric validation to ensure construct accuracy.

Authors

• Eunji Lee, University of Georgia

• George Engelhard, University of Georgia

• Zhenqiu Laura Lu, University of Georgia

• Yoojoong Kim, University of Georgia