Using BERT to Study Word Meaning Changes

• In Event: Embeddings

Abstract

Most natural language processing approaches assume that language is static. Word2vec, for example, produces a static vector for each unique word in a corpus even though many words are polysemous and may change in meanings over time. Although it should not be expected that a language model captures every meaning of a given word in a representation, it is important for political science research to consider the issue of word usage change. For example, what the word "Trump" meant before and after the 2016 U.S. general election changed overnight.

BERT produces contextual embeddings, which makes them suitable for words that change in usage over time. But unlike word embedding methods like word2vec, BERT embeddings are not linearly related to each other. This makes comparing BERT embeddings across time difficult. There is no current method that substantively use BERT to understand how word usage changes. I develop a twin neural network approach using a contrastive loss function. The contrastive loss function is used to predict whether the two sentences or documents come from the same time period or not. Because the BERT embeddings are directly compared in this approach, the outputted BERT embeddings from a twin neural network are comparable using similarity functions such as the cosine similarity function. This allows us to quantify how similar or dissimilar two sentences or documents may be.

I apply this method to two datasets. First, I use the text of the user biographies of Twitter users that tweeted an observation about the 2016 U.S. general election or the 2018 U.S. midterm election to study how usages of partisan words, such as "Trump," "Democrat," and "Republican," change between the two election cycles. Second, I use this method to see if usages of partisan words change after a factory closure, such as the closure of the GM Assembly Plant in Lordstown, OH. This application is part of a larger study on changes in interactions with alt-right and hate groups on social media in the aftermath of a factory closure.

Author

• Patrick Y. Wu, New York University