InCorporation: Assessing Corpus Selection for Social Science Applications

• In Event: Comparative Politics Applications of Latent Variable Models

Abstract

What makes a corpus a corpus? Corpus selection undergirds any inferences drawn from natural language processing projects, yet current practice treats corpora as given. Rather, "corpusness" can be considered a latent construct, with uncertainty about whether the documents under consideration either encompass the universe of relevant documents or serve as a representative selection. Corpus selection can encompass three separate problems. First is a potential detection problem: can we automatically distinguish between documents that principally belong to a corpus and those that do not? For example, a text processing task seeking to evaluate the collected writings of a single author may confront unsigned or anonymously written documents, requiring an analysis of authorship to determine inclusion. Likewise, documents might reasonably be selected on the basis of other metadata (e.g., date, language) that are detectable but unknown ex ante. Second is a question of corpus cohesion: to what extent do documents within a proposed corpus tightly coalesce around a set of concepts or topics that relate to the task or research question? For example, shifts in concept measurement (e.g., "democracy" since 1800) or changepoints in discourse plausibly suggest the existence of multiple different latent "corpora" rather than a single tightly cohering corpus evidencing a shift in concept. Finally, corpus selection entails a question of universality and representation: how can we determine whether we have selected all or the correct, "representative" set of documents that pertain to our research question? King, Lam, and Roberts (2017) highlight this challenge with respect to keyword selection of documents, but leave unanswered the theoretical question of how to assess corpus selection absent keyword determinants or even among possible keyword sets. All three of these problems suggest the need for evaluating "corpusness" to effectively weight results of text analysis to encompass uncertainty about the selection of documents into the corpus at the outset. This paper provides an initial method for distinguishing among potential sub-corpora within a selected possible "corpus," with a proposed extension to the selection of documents into possible corpora. Validation data and examples illustrate the importance of this method in particular for comparative politics applications, where case selection intersects with corpus inclusion criteria and quality.

Author

• Sirus Bouchat, Northwestern University