Record Linkage with Text: Merging Data Sets When Information is Limited

• In Event: New Directions in Political Methodology

Abstract

The recent years have seen the emergence of new, more scalable ways to link information about individuals across multiple data sources. However, merging data sets when the number of variables used for record linkage is restricted remains challenging. In this paper I consider the case when the information is limited to a single multi-token text string. This situation often occurs when researchers work with organisation names, user accounts or any other short labels. In this paper I run a simulation study showing the limitations of the existing approaches that under-perform due to a lack of standardisation and short length of text. Despite these caveats, some information, such as probability of encountering different words remains unexploited by the currently available methods. I propose an alternative approach, where the constituent tokens are weighted by their overall language frequencies. This method is applied to simulated data, as well as large number of government transparency reports from the UK. The proposed weighting method offers a more robust and scalable approach both to deduplicating the transparency reports and linking them to organisation-level covariates.

Author

• Tom Paskhalis, London School of Economics and Political Science