Synthesizing Crossnational Research: A Model Space for Democracy and Growth

• In Event: Measurement in Surveys

Abstract

Individual studies can be misleading. This is evident in that we often find replicating published work difficult and in the diversity of findings that multiple studies (of the same subject) often reveal. Accordingly, we are well-advised to canvas the literature on a subject rather than to rely on individual studies. But, some studies have more credible designs or rely on more extensive data material than others, suggesting they should be assigned more weight, and oftentimes it is very important to figure out exactly why results diverge across studies. The question arises, what is the best way to learn from a literature in its entirety? How can it be synthesized in a parsimonious fashion?
Here, we focus on the relationship between democracy and growth, a question that has attracted a great deal of scholarly attention over the years and exemplifies many of the characteristic features of observational data analysis, including divergent findings.
Our approach to knowledge cumulation may be regarded as an amalgamation of traditional approaches – namely, literature review, meta-analysis, model-averaging, and robustness tests. The core idea is to construct an expansive model space encompassing all plausible tests pertaining to democracy’s impact on growth. Results of these tests are then subjected to a second-order statistical model that should reveal the impact of different features in a concise and intelligible fashion, synthesizing the results of millions of tests into a small number of consequential choices such as inclusion or exclusion of country-fixed effects or how standard errors are calculated. The analysis will also reveal the specific concatenation of features that are required in order to obtain what we shall classify (with some arbitrariness) as positive, null, or negative effects.
This protocol will help us to better understand the literature on a subject, revealing which concatenations of modeling choices have been employed and which have been largely ignored, and allowing us to systematically compare results across these subsets. It will also reveal precisely which modeling assumptions are necessary for which set of conclusions. (What do you need to believe in order to believe that democracy causes growth?) This should narrow the range of potential disagreement, offering space for more fruitful discussions centered on those features that truly matter.
Step 1 is to canvas the literature for key variables – measures of democracy and growth – as well as background covariates commonly employed in model specifications. To deal with missingness, missing values are imputed so as to produce several full samples, one stretching back to 1960 and another (with a smaller set of variables) to 1800. Step 2 is to canvas the literature for modeling choices – functional form, specifications, estimators, and standard errors. Step 3 is to run tests for each (plausible) model. This set of models includes millions of combinations varying in terms of samples (e.g., time span, imputation approaches, etc.), data sources, and modeling choices. It entails a non-trivial computational challenge, but one that can be met with cloud-based computing.
Step 4 is to make sense of the results by modeling the data generating process – that is, the process by which different modeling choices yield different estimates. To determine this, we conduct a secondary analysis in which estimates of t and standardized b are regressed against all the features that define the model space. We employ both (regularized) linear regression and random forest approaches to make sense of the model space. This allows to identify which specific design choices make a difference, as well as the magnitude and direction of their (unconditional and conditional) effects on model results.
Our approach should reduce the results of hundreds of thousands of tests into a concise framework, allowing us to understand the practical impact of all the (plausible) design choices. We thus hope that researchers will find our framework useful, and that our illustration on democracy and economic growth will be replicated and yield insights on other major research questions

Authors

• John Gerring, University of Texas at Austin

• Carl Henrik Knutsen, Department of Political Science, University of Oslo

• Andres Cruz, University of Texas at Austin