From Voices to Validity: Leveraging Large Language Models for Textual Analysis of Policy Stakeholder Interviews

• In Event: Dashboards and Machine Learning Technology for Data-Driven Decision Making

Abstract

Obtaining diverse stakeholder opinions on current policy is crucial for effective policymaking and implementation, but traditional text analysis methods are labor-intensive and time-consuming. This study explores using Large Language Models (LLMs) like GPT-4 with human expertise to analyze K-12 education policy stakeholder interviews in a U.S. state. Using a mixed-methods approach, experts created a codebook and prompts for GPT-4, achieving nuanced thematic and sentiment analysis. Results show GPT-4’s thematic coding aligned 78% with human coding, increasing to 96% on broader themes, outperforming traditional NLP methods by over 25%. GPT-4’s sentiment analysis also closely matched human expert judgment. Qualitative comparisons highlight the complementary roles of human expertise and LLMs in enhancing efficiency, validity, and interpretability of educational policy research.

Authors

• Alex Liu, University of Washington

• Min Sun, University of Washington