A Computer Assisted Search Strategy for Digital Text Archives

• In Event: Advances in Machine Learning (II)

Abstract

With declining digitization costs an unprecedented and rapidly increasing amount of unstructured historical text has been stored and made available to researchers and the wider public. These digital archives now make it possible to investigate and answer previously intractable questions, such as the spread of diseases, the dynamic character of the judicial system, and the geographical spread of concepts and technologies. However, those digital archives often have a feature hindering the application of existing computational approaches: researchers cannot access all or large chunks of the data at once; the archive can only be searched using Boolean search masks on its website or via API. We address this gap by adapting existing human and semi-automated approaches to overcome the full-text access limitations of many digital archives. We do so by splitting up the classification approach into two stages: (1) a keyword classification stage, which aims to identify documents to download using the Boolean search result patterns, and (2) a full-text classifier, which uses the full text of the downloaded documents to identify a target set. After outlining the algorithm, we present an implementation and evaluation of the algorithm in R from a research project aiming to identify the set of newspaper articles about election violence in the British Newspaper Archive each of the 20 General Elections in England and Wales between 1832-1914. Our evaluation results show that the proposed algorithm performs similar to existing computational search strategies when all text is available and equally well or better compared to human experts on a small scale task. We thus offer researchers an easy to implement, reliable, and scalable approach based on existing computational search strategies to systematically search and identify relevant documents in digital archives stored on remote servers that can only be accessed via a Boolean search mask.

Authors

• Gidon Cohen, Durham University

• Patrick Michael Kuhn, Durham University