Using Theories of Action Approach to Measure Impact in an Intelligent Way: A Case Study From Ontario, Canada

• In Event: 51.055 - Intelligent Accountability: An Evidence-Based, Trustworthy, and Persuasive Alternative to the Status Quo

Abstract

Purpose: The aim of this paper is to present an intelligent framework designed to support the impact and scale-up of effective innovations: a key aim that should be both supported by and underpin any approach to intelligent accountability.
Perspectives: Now more than ever before, there is an impetus for schools to be more effective for more students: being able to meaningfully and effectively measure impact is thus vital. As such, researchers, practitioners and policy makers should be regularly investigating whether innovative initiatives have met their intended outcomes, and if they haven’t, assessing why this is the case. At the same time, if innovations have been found to be effective then it is reasonable to expect that these new approaches should be scaled up and rolled out across other schools, school boards and school systems that might benefit. In fact, both types of situations require educators to understand a core set of knowledge: what components of these innovations were key in enabling them to achieve their required impact? Why was this the case? And do these key components hold elsewhere? Given the drivers for schools and school systems to identify what is effective and to ensure all schools are able to tap into ‘what works’; and in light of the similarities in the knowledge required to understand whether something was effective and how others can benefit from its use, it would seem that in networked and self-improving education systems we both can and should be linking together ways of measuring the impact of innovations with the means to enable their scale-up (Earl and Timperley, 2015; Munby and Fullan, 2016).
Data source and mode of enquiry: The Dialogic Model of Impact (DMI). DMI was designed with the specific purpose of evaluating the impact of Renfrew County Catholic District School Board’s roll-out and expansion of the Through their Eyes: Documenting Literacy and Learning in Kindergarten approach to teaching and learning; as well as identify the potential value of Through their Eyes for other Ontario school boards. The development of DMI began with an exploration of current impact measurement models. But the DMI approach also stems from the knowledge that impact can and does change over time; meaning that measuring impact using purely linear approaches (e.g. simply by measuring impact as a snapshot of change) will be less effective than approaches that examine impact dynamically, and from a multitude of perspectives. As such DMI employs multiple research conversations, compares the findings that emerge from these conversations and uses this data as part of the iterative process of improvement.
Results: This paper will present both the model and detail its application, drawing on interview data from district leaders, 10 principals and focus groups of 20 teachers.
Scholarly significance: The paper will also show how DMI has general applicability and spotlight its potential use in terms to measure the impact of any innovation and to identify the components of innovations that are vital to their successful scale-up

Author

• Chris Brown, University of Portsmouth