The Necessity of Construct and External Validity for Generalized Causal Claims

• In Event: Experiments in Methodology

Abstract

The credibility revolution has facilitated tremendous progress in the social sciences by advancing design-based strategies that rely on internal validity to deductively identify causal effects. We demonstrate that prioritizing internal validity while neglecting construct and external validity prevents causal generalization and misleadingly converts a deductive claim of causality into a claim based on speculation and exploration -- undermining the very goals of the credibility revolution. We develop a formal framework of causal specification to demonstrate that internal, external and construct validity are jointly necessary for generalized claims regarding a causal effect. If one lacks construct validity, one cannot assign meaningful labels to the cause or to the outcome. If one lacks external validity, one cannot make statements about the conditions required for the cause to occur. Re-balancing considerations of internal, construct and external validity via causal specification preserves and advances the intent of the credibility revolution to understand causal effects.

When a researcher claims to have established causality with data, the researcher must undertake potentially risky inferences beyond what is implied by the data, and internal validity can address only a subset of the inferential risks. One risk concerns the proper specification of the causes and the effects. Another risk concerns properly specifying the scope conditions of the generalization. We argue that the first of these two risks can be understood as the risk of a failure of construct validity and the second can be understood as the risk of a failure of external validity.

The methods proposed by the credibility revolution identify causal effects in terms of measured variables, rather than in terms of the actual causes and effects, and often leave the scope of the generalization as implicitly local to the experimental setting. However, researchers virtually never interpret their (implicitly local) findings in terms of the measured variables themselves. Instead, they typically state a generalization that posits the actual causes and effects. When researchers make the move from measured variables to causal claims, ignoring construct and external validity undermines the intended deductive nature of the causal claims. Instead, the stated causal claims only offer speculative and exploratory ideas to account for the causes and conditions that are responsible for an observed statistical relationship among measured variables.

In this sense, the identification assumptions advanced by the credibility revolution are inadequate to support generalized causal claims. By focusing on internal validity, these methods ignore the risks for causal generalization addressed by construct and external validity even when the target estimand is identified with the data. Our formal framework, which we call causal specification, demonstrates the joint necessity of internal, construct, and external validity for making causal generalizations. We show that even a minimal deductive claim about what causes what cannot rely exclusively on internal validity. We show instead that augmenting identification with considerations of construct and external validity can preserve the intended deductive nature of the claim. Doing so is crucial to preserving the intent of the credibility revolution to focus on understanding causal effects.

See the paper at https://osf.io/2s8w5/

Authors

• Kevin M. Esterling, University of California, Riverside

• David Brady