Considerations for Pre-analysis Plans for Adaptive Experiments

• In Event: Advances in Experimental Design

Abstract

I discuss considerations for writing pre-analysis plans for researchers conducting adaptive experiments.

Experimental researchers in the social sciences sometimes face the problem of assessing multiple competing treatment interventions, where ex ante they have limited information about outcomes under the different interventions, but ex post they are most interested in the intervention that will turn out to be the most effective. Response adaptive algorithms, which dynamically update treatment assignment mechanisms based on observed response, facilitate experimental designs where the most data is collected about the most effective intervention. Algorithms can be selected to minimize regret, to optimize best arm identification, or to most efficiently produce treatment effect estimates with respect to the best arm and a control condition.

While adaptive experiments are built on the premise of designs that change as the data is collected, these changes are fully enacted by an algorithm, which is fixed by the researcher before the start of the experiment. However, as the researcher does not know how much sample will be allocated under each treatment, or with what probabilities treatment will be assigned, standard features of pre-analysis plans such as power calculations can not be implemented as they would be under conventional static experimental designs.

Instead, researchers can use carefully designed simulations to demonstrate how the experiment could play out under alternative versions of the world. These simulations can help researchers determine the necessary experiment size and can inform algorithm parameter selection. Documenting these simulations and design decisions are important to research transparency and will help researchers to design the best experiments for their objectives.

Author

• Molly Rachael Offer-Westort, University of Chicago