The Policymaker's Problem

• In Event: Generalizability in Experimental Political Science

Abstract

Some methodological approaches to external validity (or generalization) are projective. Specifically, they formulate external validity as a property of a mapping from some source, such as a grand population, to destinations in which studies are conducted. Thought of in this way, questions of external validity mirror, albeit informally, discussions of how a policymaker uses evidence from a study (experiment) conducted in one setting to inform their choice of policy in a different setting. Since this is a central goal of evidence-based policymaking, it is important to understand its theoretical foundations, thereby articulating the policymaker's problem. We formalize projective validity as one form of external validity and contrast it with others. We then embed projective validity into a decision-theoretic model of a policymaker's use of evidence to set policy. This formulation highlights the relationship between methodological issues related to external validity and a policymaker's substantive policymaking objective. Our results show when evidence-based policymaking is incentive-compatible for the policymaker, and highlight how the policymaker's own formulation of external validity influences policy outcomes.

Authors

• Tara L Slough, New York University

• Scott Tyson, University of Rochester