Utilizing Artificial Intelligence Models in Loneliness Item Evaluation

• In Event: AERA Roundtable Session Wednesday 12:40 pm Four Seasons Ballroom 2-3
  In Roundtable Session: Artificial Intelligence in Education: Innovations in Assessment, Literacy, and Performance Analysis (Table 5)

Abstract

Self-report instruments are vital in psychological research. Yet, developing such instruments is labor-intensive, requiring initial item writing, and expert review. Large Language Models (LLMs) can streamline this process through the preliminary review of items. This study examines the ability of OpenAI, Google, and Anthropic models to evaluate 28 self-report loneliness items. Results show ChatGPT-4 outperformed all other models in identifying bias and complexity, but all models struggled with idiomatic and double-barreled language and items measuring a closely related construct. While additional prompt development should deliver more accurate results, these LLMs should be seen as a complement to expert item review. Future research should focus on concise, clear prompts exploring a variety of constructs to expedite the development of new instruments.

Authors

• Joshua Spieles, University of Toledo

• David Dueber, University of Toledo

• Michael D. Toland, University of Toledo

• Allyson Givens, University of Toledo

• Falynn Thompson, University of Toledo