Informing Complier Average Treatment Effects with Post-treatment Variables

• In Event: Causal Inference: Post-treatment Bias and Conjoint Experiments

Abstract

This project proposes a new methodological framework to estimate the Complier Average Treatment Effects, when a researcher only observes a crude and biased measurement (“proxy”) of the treatment take-up status. Existing method is either to estimate a different quantity of interest, Intention-to-Treat effect, or to instrument the proxy, which I show is not applicable in the experimental setting where the instrument and measurement errors are correlated. This new method considers whether a unit is a complier as a latent variable, and then estimate the probability of a unit being a complier with a Gaussian mixture model. Simulations and the empirical example show the proposed method is working well. The most wide applications of the experimental setting covered by this new method are online and field experiments with manipulation checks, either factual or instructional. On the other hand, timers in online experiments could also be viewed as a useful manipulation check variable. This method has also discovered a new way of using these timing variables to inform the estimation of treatment effects, what existing literature has largely ignored.

Green et al (2018), studying how scandal changes voter’s evaluations of the public official involved, randomly distribute specially edited investigative newspapers reporting scandals to voters, and then collect subjective evaluation of the voters for these public officials. They compare the differences in means of the subjective evaluations between voters who have been mailed the special newspaper and those who have not.

However, what these comparisons in means reflect are the average change of a subject's evaluation caused by being mailed a newspaper rather than the average change of a subject's evaluation or belief after she has read the newspaper. That is, an Intention-to-Treat (ITT) effect, not an average treatment effect (ATE).

Researchers may use the instrumental variable approach (Angrist et al, 1996) to estimate the Complier Average Treatment Effect, a version of ATE, in these experiments. However, one of the important yet unwritten assumptions for this instrumental variable approach is that researchers can observe and record whether a subject has taken up a treatment. However, in a lot of field experiments or even online experiments, the accurate information on whether a subject has eventually taken up the treatment is not easy to obtain. Researchers have no way to know whether the research subjects have read a newspaper in their mailboxes. Usually, all what researchers can do is to put a manipulation check in their experiments. For example, Green et al (2018) asks the voters whether they could recall a scandal story about a public official.

Some researchers tend to consider these manipulation check results as a crude measurement or “proxy” of the treatment take-up status. However, there are two reasons preventing inclusion of these manipulation checks into estimation of causal inference. First, they are usually generated after the treatment has been received and thus are post-treatment. Simply conditioning on post-treatment variables for the analysis of treatment effects can bias the estimation in experiments (Montgomery et al, 2018). Second, instrumental variable approach only eliminates measurement errors in cases where the measurement error is uncorrelated with the instrument (Pischke, 2007), but in our case it is not.

This project proposes a framework to inform the estimation of (Complier) Average Treatment Effect or CATE with post-treatment variables such as manipulation checks. This method will be particularly useful when researchers cannot directly observe the treatment status of a subject but only have a crude and biased post-treatment measurement (“proxy”) of it. The framework extends the latent variable perspective framework for a canonical instrumental variable set-up proposed by Zhang et al (2009), and it also provides a new solution to the existing debate on whether manipulation check results should be conditioned on in the analyses of experiments (Berinsky et al, 2014; Montgomery et al, 2018; Kane and Barabas, 2018).

The paper proceeds as follows. I start with a formalization of the quantities of interest, and then discuss why existing methods have not provided a satisfactory estimation of the Complier Average Treatment Effect. I later propose my methodological framework for the analyses of this type of causal relationships. Finally, I show a simulation and an empirical application of the new method to show the validity of the new method, followed by discussions and plans for the next step. The framework extends the latent variable perspective framework for a canonical instrumental variable set-up proposed by Zhang et al (2009), and it also provides a new solution to the existing debate on whether manipulation check should be included in the analyses of experiments (Berinsky et al, 2014; Montgomery et al, 2018; Kane et al, 2018).

Author

• Shiyao Liu, Massachusetts Institute of Technology