Using a Probabilistic Model to Assist Merging of Large-scale Data Sets

• In Event: Big Data and Machine Learning

Abstract

Since social science research typically relies upon multiple data sources, merging data sets is an essential part of workflow for most researchers. However, if a unique identifier that links data sets is missing and, if data sets have many observations, some of which may contain missing and inaccurate information, it becomes no longer possible to combine data sets "by hand" without sacrificing the quality of the resulting merged data set. This problem is especially severe when merging large-scale administrative records such as voter files. The existing algorithms to automate the merging process do not scale, result in many fewer matches, and require arbitrary decisions by researchers. To overcome this challenge, we develop a fast algorithm to implement the canonical probabilistic model of record linkage for merging large data sets. Researchers can then combine this model with a small amount of human coding to produce a high-quality merged data set. The proposed algorithm can handle millions of observations and properly account for missing data. We conduct simulation studies to show that the proposed algorithm performs well in a variety of practically relevant settings. Finally, we use our methodology to merge the campaign contribution data, the Cooperative Congressional Election Study data, and the nationwide voter file.

Authors

• Ted Enamorado

• Kosuke Imai, Princeton University