An Embedding-Based Pre-Validation Tool for Scale Development

• In Event: Artificial Intelligence Applications in Psychometrics: From Scale Development to Response Analysis

Abstract

Objectives:
With advancements in artificial intelligence, particularly natural language processing (NLP), scale development can be simplified. Embedding techniques vectorize text, capturing semantic relationships between items, aiding in psychometric evaluation and validation.

Methods:
We use sentence-BERT models to vectorize text and compute similarities. This study is conducted based on validity-related theories, encompassing various forms of evidence including convergent validity, discriminant validity, content validity, criterion validity, and predictive validity.

Results:
The lightweight sentence-BERT models showed varying degrees of validity support, demonstrating the feasibility of embedding techniques in the pre-validation stage of scale development.

Conclusion:
Embedding-based pre-validation tools enhance scale development efficiency and quality by providing systematic early-stage evaluations.

Authors

• Shicong Feng, Peking University

• Zhehan Jiang, Peking University