Alignment Between Factor Names and Item Texts in Psychological Assessments

• In Event: AERA Roundtable Session Saturday 9:45 am - Gold 4
  In Roundtable Session: AI in Assessment and Reasoning: Innovations in Measurement and Scale Development

Abstract

In psychological assessment development, factor names are generally decided by a small group of experts. Misalignment between the factor name and the corresponding item contents/texts can occur when the factor name is too broad or narrow for the item contents, potentially leading to misinterpretation of assessment results. This study proposes an LLM-based approach to evaluate alignment between factor names and item texts. Using the psychological scales from PISA 2022, we start with fine-tuning an LLM so it can be more sensitive to capture the meaning of factor names and item texts in psychological assessments. Then embeddings for the item texts and factor names are yielded by the fine-tuned LLM. Cosine similarity is calculated to quantify the alignment.

Authors

• Mingfeng Xue, University of North Carolina - Greensboro

• He Ren, University of Washington

• Yunting Liu, University of California - Berkeley