Evaluating Topic Modeling Techniques for Chinese Political Discourse Analysis: a comparative study of BERTopic and FASTopic

• In Event: 50260 - Section on Asia and Asian America Roundtables
  In Refereed Roundtable (90 minute): Table 5: Methodological Innovation and Challenges in Social Inquiry

Abstract

Topic modeling has become a popular computational tool for analyzing political texts efficiently. Nonetheless, while breakthroughs in natural language processing algorithms have provided powerful tools to capture topic features in large corpus more accurately, there has not been a universal and clear benchmark for researchers to evaluate the performance across different topic models in the social science context. This study addresses this gap by comparing state-of-the-art topic modeling techniques applied to The Governance of China (Volumes 1–4), a comprehensive collection of Xi Jinping's political speeches. By comparing these topic modeling techniques, this research explores whether simple metrics can help us compare different models' effectiveness in uncovering policy trends. Our pilot study focuses specifically on BERTopic and FASTopic, assessing their relative strengths in the analysis of Chinese political discourse. Results indicate that FASTopic achieves superior topic delineation with a perfect Topic Diversity (TD) score of 1.0, while BERTopic produces a more moderate TD score of 0.66 but better captures conceptual relationships between topics. These findings highlight key trade-offs: FASTopic excels in isolating distinct policy themes, whereas BERTopic provides a more interconnected representation. This comparative framework contributes to the development of more appropriate evaluation standards for topic modeling in political text analysis.

Author

• Yehong Deng, University of Washington