Short-term Outcomes for Long-term Goals in Algorithmic Refugee Assignment

• In Event: Methodological Advances in the Politics of Immigration and Refugees: Part 1

Abstract

The algorithmic geographic assignment of refugees and asylum seekers is an area of algorithmic decision-making that has seen an increasing amount of research in recent years and has already been implemented on a limited basis in the United States and Switzerland. As national resettlement programs seek to improve the post-arrival outcomes of ever-increasing flows of refugees and asylum seekers, the basic idea behind this approach is to use machine learning models and optimization methods to match families to those locations within a host country where they are most likely to thrive, according to specific metrics of integration success. This is done by using data on prior refugee arrivals to train the models, which predict the integration metric in each possible location as a function of refugees' background characteristics. In considering and implementing algorithmic refugee assignment, policymakers must choose their metric of success. That is, they must choose both their integration measure of interest (e.g. employment) and the time frame/duration for that measure (e.g. two years after arrival). These are clear and known decision points that must be guided by public policy considerations. This paper is focused on a less recognized additional decision point, and one that must be guided by statistical considerations and analysis. Specifically, conditional on the integration metric and duration of interest having been chosen (i.e. the "policy outcome", such as employment two years after arrival), a subsequent decision must be made on the particular outcome variable to actually use to train the machine learning models underlying the algorithmic decision-making system (i.e. the "target outcome"). At first glance, it would seem natural to designate the target outcome to be the same as the policy outcome. However, the longer the target outcome duration, the older the data that is available to train the models (e.g. if focusing on two-year employment, the most recent training data that can be used must be at least two years old). The problem with such a lag is that the underlying relationship between refugees' integration prospects (e.g. likelihood of employment) in each location as a function of their characteristics is likely to be highly non-stationary due to economic, social, and political shifts occurring across geographies. Given the general policy interest in longer term outcomes, this non-stationarity limits the practicality of using the policy outcome as the target outcome. Instead, as is shown in this paper, it is possible that improving the longer-term policy outcome is actually most efficaciously achieved by targeting a shorter-term version of the outcome. That is, it can be most effective to target the "wrong" outcome in order to improve the "right" outcome. In addition to presenting the problem of target outcome choice, a key contribution of this paper is in highlighting the non-obvious fact that improving a longer-term outcome is, in some cases, best achieved by making decisions on the basis of a shorter-term outcome. (This stands in contrast to the typical view and usage of "surrogate" outcomes, which are used out of necessity when the actual outcome of interest would be preferable but is not available or measured.) Furthermore, the conditions under which this is more or less likely to be the case are also discussed, along with empirical strategies for making this determination and engaging with additional fairness considerations. Finally, the paper also provides an empirical application using administrative data on asylum seekers in the Netherlands, illustrating a real-world example of this phenomenon. In other words, the application demonstrates that the problem posed in this paper is not merely a theoretical novelty, but rather a real-world practical issue that should be broadly considered by designers of any algorithmic decision-making system.

Authors

• Kirk Bansak, University of California, Berkeley

• Elisabeth Paulson, Stanford University