When Does NLP Help? Human-AI Collaboration in Public Opinion Topic Modeling: A Study of E-Scooters

• In Event: Leverage Artificial Intelligence to Speed Research and Evaluation Projects

Abstract

There is a burgeoning policy literature on using natural language processing (NLP) to analyze public opinion, including the use of NLP to conduct topic modeling. Existing research has paid less attention to the people who code public opinion and how they interact with NLP tools. We study two questions that are less explored in prior studies: First, is human pre-coding useful, and to what extent does it improve the performance of NLP-based topic modeling? Second, does coders’ substantive expertise affect the performance of topic modeling?       

We conduct a case study of public opinion on the introduction of e-scooters within the built environment of university campuses across the U.S. We collect data on public opinion from Reddit, including 933 posts from 50 university subreddits. We recruit 40 coders with a background in policy analysis, including (but not limited to) those with substantive expertise in urban planning, and ask them to “collaborate” with ChatGPT to summarize topics in public opinion on e-scooters and the campus built environment. We first conduct topic modeling using an unsupervised learning approach, in which ChatGPT generates topics without human pre-coding. We then conduct a semi-supervised analysis: we ask coders to pre-code 20 percent of assigned posts, provide their topics to ChatGPT, and, for each remaining post, ask ChatGPT to (a) classify whether the human-generated topics adequately represent the post, and (b) generate additional topics if needed. We implement a third, reference approach based on supervised learning: ChatGPT classifies posts using the human-generated topics but does not generate new topics. We include two additional senior coders with an urban planning background who analyze all posts; their coding serves as the ground truth for evaluation.       

In the evaluation phase, we determine matches between the ground truth and topics generated by coders with ChatGPT based on the cosine similarity. We find that, overall, semi-supervised learning yields the highest precision, recall, and F1-score. Most coders—although not all—report semi-supervised learning as their preferred method. Supervised learning only for topic classification achieves high precision but low recall, suggesting some topics are missed due to limited coverage of human-generated topics. Unsupervised learning yields high recall and moderate precision. While unsupervised learning performs better than supervised learning, some coders express dissatisfaction with the specific wording chosen by ChatGPT.       

We interview coders and conduct a statistical analysis of coding results, which present several findings relevant to research on public opinion. First, coders with less familiarity with infrastructure or built environment planning, as well as those with lower English skills, report greater benefit from NLP and are more likely to prefer unsupervised rather than semi-supervised learning; however, their performances remain the weakest, as measured by evaluation metrics. Second, manual coding serves as a baseline, and its accuracy significantly influences the performance of semi-supervised learning. Third, although NLP provides the least assistance to those with substantive expertise and the most assistance to those with limited substantive knowledge—measured by improvements from supervised to semi-supervised learning—coders with substantive expertise consistently achieve highest performances.

Author

• Dafeng Xu, University of Washington
  Presenting Author