Impact Measurement in the Third Sector: Policy, Science and Practice perspectives

• In Event: Impact and Performance Measurement

Abstract

POLICY: The concept of “Mission-oriented research and innovation”, which has become highly influential both on the level of the EU and – to varying degrees – of its member states, argues for a stronger role of the state in shaping innovations, by aligning their expected outcomes with missions that are deduced from the Sustainable Development Goals (SDGs) and which are expected to solve the “Grand Challenges” of our times, including climate change, anti-democratic backlash and social inequality (European Commission 2017). Innovations are to be assessed according to their positive and negative societal impacts, as opposed to being determined by their ROI. Third Sector Organizations (TSOs) are supposed to play a key role in these missions, but are expected to measure and prove their impact in a comparable, i.e. standardized manner. Impact has become a key concept in SR&I and other policy fields as well.

SCIENCE: At the same time, the theory and practice of evaluation of private and public funded programs in international development, education, social and ecological work has become more sophisticated and has moved on from assessing only the performance of organizations and their activities to measuring their outcomes and impacts (Gertler et al. 2016). The 2019 nobel prize in economics for Banerjee, Duflo & Kremer, in recognition i.a. of the application of RCTs in international development evaluation, illustrates this process. The idea of impact measurement based on comparisons with counterfactuals has inspired several quantitative and qualitative methods including RCTs, quasi-experimental designs, causal mapping and more. Additionally, researchers and organizational consultants have developed numerous process models, such as Theory of Change or the Five Dimensions of Impact. A recent meta-study of Social Impact Assessment models reviews as many as 98 different models (Corvo et al. 2021). While some efforts of standardization of models and indicators can be observed recently – which is desirable from the scientific point of view since it would provide comparable data – the field is still very diverse and dynamic, offering different (and at times: incompatible) methodological, theoretical and disciplinary perspectives.

THIRD SECTOR PRACTICE: There is surprisingly little evidence on how TSOs react to these complex opportunities and high expectations. Neo-institutionalism would suggest the hypothesis that the high external pressures and low capacities to implement actual IMM will result in a decoupling of the internal and external aspects of the organization, i.e. they will produce (potentially biased) impact evaluations to “prove” their impact, mainly to secure access to funding, as opposed to “improving” their activities to create more positive impact. In our paper, we will present available data and international studies on IMM practice in TSOs, which are very scarce, as well as results of our own, ongoing research, including a survey of social enterprises in Germany and in-depth interviews with German and European managers of TSOs. We aim to understand better the state of the art of IMM from the three perspectives of policy, science and practice, in order to allow for a realistic view of IMM’s potential and feasibility vis-à-vis the high expectations that are involved.

References

Corvo, Luigi; Pastore, Lavinia; Manti, Arianna; Iannaci, Daniel (2021): Mapping Social Impact Assessment Models: A Literature Overview for a Future Research Agenda. In: Sustainability 13 (9), S. 1–16. DOI: 10.3390/su13094750.
European Commission, Directorate-General for Research and Innovation (2017): Towards a mission-oriented research and innovation policy in the European Union. An ESIR memorandum: executive summary. Online verfügbar unter https://data.europa.eu/doi/10.2777/0956.
Gertler, Paul J.; Martinez, Sebastian; Premand, Patrick; Rawlings, Laura B.; Vermeersch, Christel M. J. (2016): Impact Evaluation in Practice, Second Edition: Washington, DC: Inter-American Development Bank and World Bank.

Authors

• Filip Zielinski, CSI Universität Heidelberg

• Georg Mildenberger, Centre for Social Investment