Evaluating Topic Models: Puerto Rican Legislators Talk Statehood on Twitter

• In Event: Civic Media: Local Government and Elected Officials' Use of Digital Tools

Abstract

One topic at the center of recent US political discourse is the possibility of Puerto Rican statehood. Mainland US news coverage of statehood focuses on how the recent Democratic takeover of the US Congress makes this potentiality one step closer to reality. Such change would, perhaps, lead to a long-term power shift securing the Democrats’ majority indefinitely because Puerto Rico would likely elect all Democrats. Largely absent from the mainland US media discourse, though, is whether Puerto Ricans actually support statehood. Even further removed from mainland discourse is the fact that the primary dividing line separating local partisan politics in Puerto Rico is drawn across support for statehood versus independence. This has never been more evident than in Puerto Rico’s recent election. There was a referendum on the ballot aimed at gauging citizen support for statehood. The referendum results were close to a 50/50 split. While the politics and consequences of this referendum are fascinating, we plan to use this context as an empirical test for the efficacy of Topic Modeling to characterize political communication on Twitter.

Latent variable modeling methods, such as latent topic models have become increasingly popular in social media political communication research. These methods are similar to factor analysis or latent class analysis in that this general family of procedures models a larger set of data to identify a smaller set of concepts. In this case, it is a corpus observed words being modeled as a function of a much smaller number of latent topics. Support for topic modeling comes from the idea that using the actual correlation between words to identify topics is a better measure than relying on traditional dictionary based methods of content analysis that are fraught with error because they rely on pre-determined lists that may mischaracterize the communication, or completely miss parts of it. Both methods make assumptions, but the latter may have more.

We contend here that the Puerto Rican case makes an ideal test of how well topic models classify words into topics. We plan to collect the corpus of Twitter communication by legislators elected to Puerto Rico’s Legislative Assembly (House = 51 seats, Senate = 27 seats) in the recent election. This makes a perfect case to evaluate topic model validity because much of the election discourse was centered on statehood, and we know exactly how the parties are divided on the issue. If this latent modeling technique works well it should classify the discussion on these topics on party lines. We will build models to determine how well the partisanship of legislators classifies communication about statehood.

Authors

• Mayra Velez-Serrano, University of Puerto Rico, Rio Piedras Campus

• Jason Gainous, Duke Kunshan University

• Kevin Wagner, Florida Atlantic University