Remote Control: Debiasing Remote Sensing Predictions for Causal Inference

• In Event: Better Models, Better Predictions

Abstract

Advances in machine learning and the increasing availability of satellite imagery have led to the proliferation of social science research that uses remotely sensed measures of human activity or environmental outcomes to infer the impact of policy. However, when machine learning models minimize a standard loss function the predictions they generate will often produce biased estimates when used for causal inference. If the prediction error in the outcome variable is correlated with policy variables or important confounders, as is the case for many widely used remote sensing data sets, estimates of the causal impacts of policies will be biased. In this paper, we show how this bias can arise using ground-truthed forest cover data in West Africa. We develop a novel approach to correct this issue by applying an adversarial debiasing algorithm.

A common use of machine learning predictions in environmental economics is to estimate the effect of some treatment on deforestation (See Balboni et al. 2022 for a review of recent work). We show that the widely used Hansen (2013) data (10,000 citations) leads to severe bias in estimating the effect of the distance to a road on forest cover. While other papers have pointed out other issues with using remotely sensed variables for causal inference (Josey et al. 2022, Fowlie et al. 2019, Ratledge et al. 2021, Jain 2020, Gibson et al. 2021, Alix-Garcia and Millimet 2020, Garcia and Heilmayr 2022, Carleton et al. 2022) ours is the first to 1) show that using land cover predictions with systematic measurement error can produce biased downstream estimates, and 2) show how adversarial debiasing can resolve the issue.

Adversarial debiasing developed within the machine learning field of algorithmic fairness, which has been used to ensure that algorithms do not make systematically inaccurate predictions on the basis of race or gender (Zhang et al. 2018). We adapt the approach to generate a satellite-derived measure of forest cover which is unbiased across the range of the treatment variable, allowing us to generate an unbiased estimate of the true effect.

Data

We use a hand-labeled dataset of 20,621 points in West Africa (Bastin et al. 2017) that are coded as forest or not forest as our "true" measure of forest cover. Our key independent variable is the distance to a road. We use data from the Landsat 7 ETM sensor in a random forest, a regularized logistic regression, and a neural network to predict the forest/not forest labels. Finally, we add control variables including slope, elevation, latitude, aspect, and others (some TBD) to our model.

Methods

Our methods section has three parts: we estimate the "true" parameter for the relationship between distance to road and the probability of forest cover. Then, we use the same method used by Hansen (2013) to generate out-of-sample measurements of forest cover for each point for which we have the true value. We then estimate the parameter of interest using the predicted forest cover values. Finally, we use the adversarial debiasing method to generate predictions of forest cover at each point. Using these predictions, which are no longer systematically biased with respect to the treatment variable, we estimate the parameter of interest.
The adversarial debiasing method minimizes the commonly used mean squared error loss while maximizing the mean squared error of an "adversary." In our case, the adversary is a second regression that predicts the distance to a road using the errors from the original regression. By maximizing the adversary's loss, we ensure that the measurement error from the remote sensing model is uncorrelated with the treatment variable. The estimates produced by the adversarial debiasing method have similar overall accuracy to those produced by traditional methods, though they are less subject to issues caused by class imbalance.

Results

We show that using the method employed by Hansen (2013) generates land cover predictions that underestimate forest near roads and overestimate forest far from roads. When used to estimate the relationship between road distance and forest, this dataset leads researchers to unknowingly double the coefficient of interest. However, when we use the debiased predictions, we recover an estimate much closer to the true parameter at only a small cost to the overall accuracy of the measurement model.

Conclusions

The use of remotely sensed outcomes in economics research is rapidly expanding, particularly in the areas of deforestation, electrification (night lights), and air pollution. However, researchers which use off-the-shelf satellite-derived measures of these key dependent variables are likely to introduce bias to the results of their inferences. We formalize the problem, identify the sources of bias, and provide researchers with an easy-to-implement method to remove this bias.

Authors

• Eliana Stone, Yale University

• Luke Sanford, Yale University

• Matthew Gordon