When problems get more complex and knowledge is fragmented: are our evaluation findings used to contribute to policy and implementation?

• In Event: Education across the globe: Facing conflicts and challenges

Proposal

This paper presents the results of a study exploring the role of knowledge in evaluation. The study specifically explores the extent to which the inclusion of disciplinary content, concepts, and constructs could influence the utilisation of evaluation findings for policy and decision-making. In doing so, the study seeks to contribute to the literature on evaluation relating to ‘theory-based’ evaluations (Coryn et al. 2011). It does this by exploring the nature of the knowledge that underpins the theory in theory-based evaluations, and the implications that knowledge has on the way in which the problematic and solutions are described in the programme theory (the subject of the evaluation), the decisions that are made related to evaluation design and in the analysis of the data. The study then reflects on the implications that this has on utilisation within the context of the evaluations of “complex” policies or programmes (Rogers 2008).


These questions, pertaining to the relationship between theory and utilisation, are considered within the context of evaluations of interventions that address the problem of youth unemployment, which is an urgent challenge in South Africa and globally, and one that can only be addressed drawing on knowledge from across multiple disciplines and through the implementation of ‘complex’ policies and programmes. The importance of addressing this imperative is reinforced by the reality that multiple players, including the African Union, African Development Bank, UNDP, ILO and the World Bank, are directing considerable resources at youth employment programmes (Fox, Senbet, and Simbanegavi 2016, 14). Yet despite the concomitant resources spent on evaluations, there are concerns about the extent to which these evaluations contribute to evidence-based decision making. Evaluation findings are not, in the main, a significant basis for government-decision making.


When reviewing the literature and drawing on 30 years of experience as an evaluator what becomes apparent is that resolving the challenge of utilisation often carries with it an over emphasis on methods-oriented approaches and conversely an absence of a focus on the knowledge component of theory-based evaluation. This disproportionate focus on methods and tools in turn leads to the focus of quality improvements being placed almost entirely on the development of evaluation as a practice and the technical skills of the individual evaluator. Further, the methods-oriented approaches reduce the discussion about quality and utilisation to a debate about which method is the most ‘scientific’. This frequently leads to reliance on what could be termed “mathematical” tools rather than on understanding the causes of the problem that is being addressed by the intervention being evaluated or on locating this within the context in which changes are taking place.


Even where there is a recognition of the importance of theory-based evaluation there is no shared understanding of what form of knowledge should constitute the theory component of theory-based evaluation. There is considerable contestation about the ‘sources’ of knowledge that are relied upon for the development of the programme theory. While in some cases there is a shift towards an appreciation of the importance of synthesising previous research and evaluation findings in order to locate new findings, these developments frequently do not consider the nature of the knowledge that needs to underpin an evaluation nor the divergent approaches to analysis (Becher 1994) that are required as part of building the programme theory. This encourages a ‘siloed’ approach to evaluations. To address this perceived gap I have turned to the work of educational sociologist Basi Bernstein (1996; 1999) and many of the scholars who draw on Bernstein such as Muller, Young and Moore (Muller, Davies, and Morais 2004; Rob Moore and Muller 1999; Rob Moore 2011; R Moore and Maton 2001; G. F. Moore et al. 2019; Young and Muller 2007; 2013; 2016). Insights from the school of writing within the sociology of knowledge highlight that engaging with the structure of knowledge may be potentially important in guiding the ways in which disciplinary knowledge can be integrated into evaluations.


In undertaking this study I adopted a number of different approaches. This study is positioned under conceptual research because it attempts to engage with how evaluation is theorized, and how theories of knowledge can interact with and support theorizing evaluation. There is also an empirical component to the research, which consists of analysis of interviews with 10 individuals responsible for policy development and decision making as well as for the evaluation of youth employment interventions. These interviews are critical to explore issues of utilisation and to reflect on the extent to which a stronger emphasis on knowledge could support the trustworthiness (validity and ability to generalise) of the findings, assisting to answer the questions of ‘what works for whom?,’ and making it easier for policy makers and decision makers to consider the implications of these findings for future interventions and policies.


The envisaged significance of this research is that it could ensure that evaluative studies interact in a meaningful way with prior research and knowledge. In doing so, the findings of this study could encourage a “two way traffic between the socialized reservoir and the practitioner repertoires” (Young and Muller 2013) or, phrased differently, between the conceptual and the empirical in order to allow a move from the contextual to the general. As an individual who has also been working on policies relating to youth employment, my hunch is that while ensuring the quality of the methods is vital for quality (with reference to validity, reliability, etc), it is only through this interplay that it will be possible to strengthen causal claims and answer questions of impact as well as support the accumulation of knowledge. This suggests that there should not be a tension between immediate utilisation and the longer term project of knowledge building, but rather that there is a need to reassert the inter-relatedness of these imperatives.

Author

• Carmel Marock, University of the Witwatersrand