A Model to Assess the Impact of Brokering Efforts: Proposed Indicators for Multi-Stakeholder Knowledge Mobilization Efforts

• In Event: 21.033 - Knowledge Brokers and Boundary Spanners: Strengthening Ties Between Research, Policy, and Practice in Education

Abstract

Purpose and Perspective
There is growing recognition of the importance of the roles of research brokering organizations (RBOs) in bridging research-practice-policy gaps in education (efforts we call knowledge mobilization, KMb); however, RBOs remain underexplored (Davies & Nutley, 2008; Nutley et al., 2007; Tseng, 2007). This study proposes impact measures that can be used to assess the efforts of RBOs in KMb processes.

Methods and Data
This paper builds a model to assess the impact of brokering efforts from a program of research that has explored research brokering and how to measure impact across three domains: research production, research use, and research mediation. Studies contributing to the model include: (1) Author’s (2014) study which mapped the efforts of 44 RBOs across Canada; and, (2) An environmental scan of impact indicators across 32 countries (Canada, the USA, the UK, the European Union, Australia and New Zealand) to answer the question: Is there a “best way” (best method) to evaluate the impacts of Humanities and Social Science research in Canada, and are there “best metrics” that could be used to assess those impacts (or improve them)? 1105 impact indicators emerged from the scan and were further categorized in relation to Cooper’s (2014) brokering functions.

Results and Significance
Author’s (2014) analysis revealed 8 major brokering functions of Canadian RBOs in education: (1) linkage and partnerships of diverse stakeholders around priority issues, (2) raising awareness about research, (3) increasing accessibility to research-based resources, (4) increasing user engagement with research, (5) capacity building among practitioners to increase research use, (6) implementation support for initiatives in school districts, (7) organizational development and, (8) policy influence.

Impact indicators proposed include eight categories which mirror brokering functions: (1) collaboration indicators: #products/ services developed or disseminated with partners, social network growth (assessed by social network analysis); (2) reach indicators: # distributed, # requested, # downloads/hits, media exposure; (3) usefulness indicators: read/browsed, satisfied with, usefulness of, gained knowledge, changed views; (4) use indicators: # intend to use, # adapting the research; (5) practice change indicators: # or type of capacity building efforts, # training and education sessions, commitment to change, observed change, reported change; (6) program and service indicators: fidelity and uptake, documentation and feedback, process measures; (7) system indicators: infrastructure, strategic planning across provinces or states, annual reporting structures; (8) multi-level indicators: # used to inform policy/advocacy/enhance programs, citations in policy, invitations of researchers to meetings, involvement in policy process, media presence. Figure 1 presents the model to assess brokering activities of multi-stakeholder KMb networks integrated from these two studies.

This research will extend our knowledge about: what strategies intermediaries utilize to accomplish their mandate to better connect research, policy and practice; any available evidence on the effectiveness of these strategies; what impact indicators can be used to measure the efforts of RBOs; the relative merits of different strategies in relation to diverse target audiences; and, why some intermediaries (and the stakeholders they serve) are more effective at KMb than others.

Authors

• Amanda Cooper, Queen's University

• Stephen William MacGregor, Queen's University - Kingston