Measuring Meaning: Using Cross-Encoder to Estimate the Similarity of Short Texts

• In Event: Methodological Developments in Text-as-Data

Abstract

In many settings, scholars wish to estimate the similarity of political texts. For instance, legislative studies measure the similarity of bills and laws to trace the flow of policy ideas, quantify legislative effectiveness, or investigate the adoption and diffusion of policies. Scholars of political communication analyze the similarity of texts created by different actors to understand information distortion on social media or the responsiveness of public speeches to public opinion. However, the most commonly used methods in political science struggle to identify when two texts convey the same meaning as they rely too heavily on identifying words that appear in both documents. This limitation is especially salient when the underlying documents are short, an increasingly common use-case. Short texts are by definition not many words, and therefore word overlap is infrequent. Moreover, short texts can easily be paraphrased using different words or happen to use a similar vocabulary while having entirely different connotations. Building on recent advances in computer science, I propose a meaning-based approach to precisely estimate the semantic similarity of short text pairs using cross-encoder. The specialty of cross-encoder model lies in processing a pair of documents together to extract feature representations. These representations are then fed forward into a neural network to classify documents in terms of similarity. Scholars can either use pre-trained cross-encoders or train a customized model with existing open-sourced tools. I illustrate this approach in two examples. First, I estimate the similarity of Facebook posts from members of Congress and test the theory that elites are less polarized when discussing international compared to domestic policy issues. In the second example, I apply the model to predict whether two news headlines about the same case are semantically similar and find that court decisions accompanied by dissenting opinions are more likely to receive different media framing.

Author

• Gechun Lin, Texas A&M University