Measuring Policy Outcomes: Data-Driven or Substance-Driven Scales?

• In Event: Poster Session: Political Methodology

Abstract

Scholars have increasingly turned to “data-driven” ideal point estimation techniques such as Bayesian IRT to estimate ideal points based on roll-call votes (Clinton, Jackman, and Rivers 2004), mass preferences (Bafumi and Herron 2010; Tausanovitch and Warshaw 2014), campaign contributions (Bonica 2013, 2014), and, more recently, policy outcomes (Tausanovitch and Warshaw 2014; Caughey and Warshaw 2015). The most appealing characteristic of such techniques is the promise of assumption-free estimation of ideological dimensions. There has been a growing chorus, however, that cautions against interpreting such scales as measures of ideology (e.g., Lee 2009; Bateman and Lapinsky 2016; Broockman 2016; Broockman and Ahler 2016; Grimmer and Powell 2016).

In this study, I argue that validity problems are especially acute for the data-driven measurement of policy outcomes. Such scales outsource the determination of the ideological content of policies to the clustering and uniqueness of policy outcomes across polities and time—which is neither normatively neutral nor substantively interpretable. I find that a recent measure of “state policy liberalism” (Caughey and Warshaw 2015, hereafter SPL) both produces substantively invalid discrimination parameters for policies and leads to substantive conclusions that dramatically contradict the state policy literature.

The discrimination parameters of policies in the SPL model are determined by the idiosyncratic historical record of policy passage across states and time. As a result, the parameters make little substantive sense in direction or magnitude. State occupational licensing policies point in different directions (e.g., extreme and liberal for dentist licensing, but extreme and conservative for nurse licensing). The corporate income tax receives a large negative (conservative) parameter. Policies that legalize abortion pre-Roe v. Wade have little effect on states’ ideal points because states like Mississippi (a state with conservative racial and labor policy) were the first to pass them. Medicaid funding for abortion receives a discrimination parameter over twenty times larger in magnitude than that of legal abortion itself. Socioeconomically inconsequential policies, such as those mandating registration of beer keg rentals, receive parameters of greater magnitude than deeply consequential policies like forced sterilization of the mentally disabled.

I then engage in an alternative measurement strategy: simple weighted averages, with sets of policy and issue area weights determined ex-ante by the researcher. Accordingly, I create substantive scales based on clear normative determinations, and also simulate scales based on randomly generated weights. Unlike roll-call votes and survey responses, datasets of policy outcomes typically contain fewer than 150 policies—and researchers are highly capable of substantively justifying parameters for each of such medium-N sets of policies.

These substantive measures have two major advantages. First, substantive conclusions drawn from their estimates conform better to those of qualitative and policy scholarship. Whereas Caughey, Warshaw, and Xu (2016, 1) use the SPL scale and find that in recent years the “effects of party control have approximately doubled in magnitude,” I find that they have more than quadrupled. Moreover, the substantive scales show that the effect of unified Republican control of state government has increased dramatically since 2000. In contrast, the SPL scale shows no increase in the effect of Republican control since 2000 (Caughey, Warshaw, and Xu 2016), a conclusion that diverges radically from recent state policy scholarship (e.g., Kersten 2011; Jacobs 2013). The SPL estimates also suggest that policy in Wisconsin in 2014 was more liberal than in 2008, and Minnesota and Wisconsin are more similar in state policy liberalism in 2014 than they were from the mid-1990s through the mid-2000s. The substantive scales, in contrast, show a major divergence between Wisconsin and Minnesota in the post-2010 Scott Walker era.

Second, the substantive scales can be easily disaggregated into multidimensional issue area scales, whereas Bayesian IRT models must be reestimated with completely different parameters. This allows me to estimate party polarization in each issue area. I find that nearly every issue area has polarized dramatically since 1970, but with a major and substantively consequential exception: the punitive criminal justice policies that led to mass incarceration. I validate this finding with analyses of the effect of party control on incarceration rates.

This study suggests that while data-driven latent dimension estimation techniques have generated considerable insight, they may not be appropriate for summarizing policy outcomes. I caution against the use of the SPL model’s ideal points as a “plug-and-play” dataset for substantive researchers.

Author

• Jake M Grumbach, UC Berkeley