No Need for a Panel Study: Measuring Moderators Doesn’t Alter Treatment Effects

• In Event: Experiments in Methodology

Abstract

As experiments have become more popular in public opinion research, scholars are increasingly interested in who is most responsive to a treatment. For example, a researcher may want to investigate who is most likely to follow a partisan cue, such as strongly identified partisans, providing more insight into the nature of elite influence over public opinion. However, there is an ongoing debate about where to place moderating variables within a survey. Researchers worry that placing a moderator prior to an experiment may 1) directly influence the treatment effect and/or 2) shape the strength of the relationship between the treatment, the moderator, and the outcome. Returning to the previous example, measuring a respondent’s party identification directly before a treatment that includes a party cue may strengthen partisan considerations, exaggerating the effect of the party cue, particularly among those who hold a partisan identity that has just been primed.

There are two alternative approaches to measuring moderators: placing it after the experiment or placing it in a prior wave of a panel survey. However, recent work makes clear that placing a moderator after an experiment risks post-treatment bias and thus should not be considered. Placing it in a prior wave virtually ensures that the measurement of the moderator will not influence treatment effects, but panel studies are costly, introduce potentially serious problems with attrition, and the moderator itself may change over time. Thus, experimenters are left without clear guidance on how to conduct moderation experiments.

In this project, we seek to clarify this debate by systematically testing whether measuring moderators within the same wave as an experiment (pretreatment) alters treatment effects. Using a series of panel studies, we replicate well-established survey experimental paradigms that use moderators such as policy attitudes, political values, racial resentment, and partisan identity. Additionally, we rely on a variety of samples, including those drawn from undergraduate subject pools, MTurk, and the Cooperative Congressional Election Study (CCES).

In each study, we measure the moderator for all respondents in the initial survey wave, then randomly assign whether respondents also receive the measure in the second, experimental wave. This design allows a clean test of whether the presence of the moderator in the experimental wave influences the substantive conclusions drawn from the study. In general, we find remarkable consistency in the relationship between moderators and treatment effect. These findings suggest that researchers need not invest in costly panel designs and potentially introduce problems with attrition. We conclude with practical guidance for researchers on how to design and conduct moderation experiments.

Authors

• Geoffrey Sheagley, University of Georgia

• Scott Clifford, University of Houston