A Large-scale Crowdsourced Re-Investigation of Immigration and Social Policy

• In Event: Poster Session: Comparative Politics in Developing Countries II & Comparative Politics of Advanced Industrial Societies
  In Poster Session: Poster Session: Comparative Politics: Advanced Democracies

Abstract

The number of foreign-born residents in rich democracies increased steadily since World War II. Scholars, pundits and the public attribute a wide list of fundamental societal and economic consequences to this and the societal and political changes it brings. One ongoing discussion concerns a popular hypothesis that immigration undermines social policy because it decreases citizen support for the welfare state. But research on this for over two decades now, is inconclusive. This study has two purposes: first, to examine the origins of differences in social scientific results obtained by different research teams. And second, to improve our understanding of the substantive question under investigation, the nexus between immigration and policy preferences.

To achieve this goal, we adopt a many analysts approach (Silberzahn et al 2018). Selecting a prominent recent study in the area of immigration and social policy we conducted a crowdsourced replication project involving nearly 200 individual social scientists participating in 88 research teams. Our crowdsourced approach improves on previous work because it includes not only many analysts, but deliberation among them, and incorporation of publicly available data in an area well researched and of scholarly and public interest. In a first step, this research involved direct replications aimed at reproducing the findings presented in the original study that used International Social Survey Program (ISSP) data (Brady and Finnigan 2014). Second, the research teams engaged in an expansion of their work still using ISSP data. In doing so, researchers were free to design the test of the original study’s hypothesis in whatever way they deemed best so long as it used the ISSP data – one of the most widely used among studies aiming to test this research question. This entirely new approach to testing for an impact of immigration on social policy preferences reduces the bias attributable to a discipline, career stage, funding source and in general any single study. It offered participants equal authorship regardless of the outcomes. It thus overcomes biases associated with the reproducibility crisis in science such as p-hacking and HARKing (Munafò et al 2017). It provides a basis for meta-analytic techniques that include input from the many researchers themselves, the principal investigators and the statistical results produced by so many of the ‘same’ studies.

Descriptively the crowdsourced results show that the original study is verified by 94% of the replications numerically. Of the expansions, 67% follow Brady and Finnigan’s conclusion rejecting the hypothesis that immigration undermines support for social policy, while 15% support the hypothesis finding a systematic negative association of higher immigration or immigration rates, 10% claim the hypothesis is not testable with these data, and 19% offer mixed results. Of the mixed findings, 67% suggest that only immigration measured as stock of foreign-born has a significant negative association, 27% suggest that only net migration does, and the remained come to some alternative mixed conclusions.

Using meta-analysis, we uncover that choices related to independent variable measurement and inclusion, the sample of countries and the type of model have small but significant impacts on the nature of results. This combined with internal deliberation and peer review by the analysts with each other leads us to conclude that our study supports Brady and Finnigan’s original conclusions in general, but raises critical theoretical points. Given that research designs more similar to theirs are more likely to support their results, we must think more about what the underlying causal model is. If including other countries in the sample affects the results, then this calls into question the generalizability of the findings. If something as simple as the inclusion of unemployment can shift the substantive conclusions we had better think very hard about whether unemployment is part of the data-generating model or not. We conclude with a summary of the main theoretical points uncovered by our research that demand further investigation.

Brady, David and Ryan Finnigan. 2014. “Does Immigration Undermine Public Support for Social Policy?” American Sociological Review 79(1):17–42.

Silberzahn, R., E. L. Uhlmann, D. P. Martin …. B. A. Nosek. 2018. “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results.” Advances in Methods and Practices in Psychological Science 1(3):337–56.

Authors

• Nate Breznau, Mannheim Centre for European Social Research (MZES)

• Eike Mark Rinke, University of Leeds

• Alexander Wuttke, University of Mannheim