Adaptive Designs for Best Arm Evaluation with Application to Vaccine Messaging

• In Event: Methodological Advances in Causal Inference

Abstract

This paper contributes to the literature on adaptive experimental designs for best arm identification. In particular, we consider a setting where the experimenter wishes not only to learn the best arm, but also to evaluate response under that arm, or treatment effects with respect to the best arm and a control condition. Adaptive designs can help experimenters more efficiently learn which treatment condition among a set of alternatives will perform best on a given response measure. However, naive frequentist estimates of response under the best arm will be upwardly biased if they are calculated on the same data used to learn which arm is best. To resolve this, we consider a two-stage design where best arm identification is learned in an adaptive first stage, and evaluated alongside a control condition in a second stage. We show improvements in simple regret, and bias and precision of treatment effect estimation as compared to alternative designs where learning and evaluation are objectives.

In our application, we design and deploy a Facebook Messenger chatbot to test messages targeting sources of COVID-19 vaccine hesitancy among 22,000 social media users in Kenya and Nigeria. After optimizing messaging using an adaptive experimental design, we compare the interactive concern-addressing chatbot to a chatbot that delivers a non-interactive public service announcement (PSA), as well as to a control, no information, condition. We find that the concern-addressing chatbot increases COVID-19 vaccine willingness and intentions by 4% in Nigeria and 5% in Kenya compared to the control condition, and by 4% in Nigeria and 3% in Kenya compared to the PSA intervention.

Authors

• Molly Rachael Offer-Westort, University of Chicago

• Leah R Rosenzweig, Stanford University