Co-Creating Impact Measures of Research Mobilization With Intermediary Organizations

• In Event: 31.052 - Advocacy Networks and Intermediaries in Educational Policy: Local, National, and Global Perspectives

Abstract

There is growing recognition of the importance of the roles of intermediary organizations (IOs) in bridging research-practice-policy gaps in education (efforts I call knowledge mobilization, KMb); however, IOs remain underexplored (Davies & Nutley, 2008; Nutley et al., 2007; Tseng, 2007). This study develops impact measures, through multi-stakeholder panels comprised of IOs and the educational stakeholders they work with, that can be used to assess the efforts of IOs in KMb processes.

METHODS AND DATA
This paper builds on [AUTHOR’S] (2014) study which mapped the efforts of 44 IOs across Canada using a matrix tool to measure KMb efforts in relation to evidence-based strategies (products, events and networks) and indicators (types, ease of use, target audience). The top 15 IOs (according to the matrix scores) were chosen to collaborate in this study, which aims to co-create impact measures of IOs with various stakeholders that they work with, including educational leaders, teachers, parents, policymakers and other community stakeholders. These 15 IOs include different types of intermediaries such as: a ministry of education (equivalent to State Education Agency in the US), advocacy organizations, research centres from universities, as well as think tanks. Multi-stakeholder panels were convened to propose an impact framework to measure their work.

RESULTS AND SIGNIFICANCE
[Author’s] (2014) analysis revealed 8 major brokering functions of Canadian IOs in education: 1) linkage and partnerships of diverse stakeholders around priority issues; 2) raising awareness about research; 3) increasing accessibility to research-based resources; 4) increasing user engagement with research; 5) capacity-building among practitioners to increase research use; 6) implementation support for initiatives in school districts; 7) organizational development; and 8) policy influence. Multi-stakeholder panels met to co-create impact measures for these 8 brokering functions. Initial indicators proposed by these panels include eight categories which mirror brokering functions: 1) collaboration indicators: #products/ services developed or disseminated with partners, social network growth (assessed by social network analysis); 2) reach indicators: # distributed, # requested, # downloads/hits, media exposure; 3) usefulness indicators: read/browsed, satisfied with, usefulness of, gained knowledge, changed views; 4) use indicators: # intend to use, # adapting the research; 5) practice change indicators: # or type of capacity building efforts, # training and education sessions, commitment to change, observed change, reported change; 6) program and service indicators: fidelity and uptake, documentation and feedback, process measures; 7) system indicators: infrastructure, strategic planning across provinces or states, annual reporting structures; 8) multi-level indicators: # used to inform policy/advocacy/enhance programs, citations in policy, invitations of researchers to meetings, involvement in policy process, media presence.

This research will extend our knowledge about: what strategies intermediaries utilize to accomplish their mandate to better connect research, policy and practice; any available evidence on the effectiveness of these strategies; what impact indicators can be used to measure the efforts of IOs; the relative merits of different strategies in relation to diverse target audiences; and, why some intermediaries (and the stakeholders they serve) are more effective at KMb than others.

Author

• Amanda Cooper, Queen's University