Transforming evidence standards into a digital tool for assessing the strength of evidence generation and use

• In Event: ENJOY: A digital framework for assessing and promoting education practitioners’ evidence practices

Proposal

Problem
Though evidence frameworks may provide a useful conceptual base for establishing what “good evidence” looks like, they may be difficult for education practitioners and a non-research audience to use for assessing evidence practices or supporting evidence-informed decision making.

Solution
Promoting utilization of evidence frameworks requires easy-to-use tools that show clearly how the definitions and concepts proposed by evidence frameworks apply to specific evidence practices, produce accessible and easy-to-interpret “diagnostic” data, and can be used broadly within and across organizations. The ENJOY digital assessment tool seeks to address this need by translating evidence standards (proposed in a previous presentation) into a tool designed to help organizations: (1) take stock of their evidence culture and (2) examine the strength of the evidence base supporting the effectiveness, transferability, and scalability of particular interventions. ENJOY upholds best-practice approaches for designing self-administered surveys (e.g., simple language, seamless navigation, extended definitions) and supports interpretation of results by providing users with a summary graphic showing which of their evidence practices are advanced, require further support, or do not apply to their current work. Eventually, ENJOY will connect users with resources that can help them strengthen their capacity and skills in the relevant areas.

Approach
This organization (a research and data analytics consultancy acting as learning partner to the foundation supporting ENJOY) developed ENJOY by leveraging expertise in research surveys and digital learning tools. In the tool’s initial stage of development, a team of survey experts transformed the evidence standards into a two-part questionnaire comprised of 12 questions. The first part of the questionnaire asks respondents to rank their organization’s evidence culture. The second asks respondents to assess the evidence base (considering effectiveness, transferability, and scalability) of a particular program or intervention. The responses to all questions are designed to rank organizations’ practices and/or evidence base on a qualitative scale (nascent, promising, established, advanced). Once a draft questionnaire was in place, the team revised it to improve its simplicity and clarity. The team also added definitions and resources that would be accessible to respondents. ENJOY’s second stage of development involved transforming the questionnaire into an easy-to-use digital survey with built-in data visualization capabilities. The tool’s primary visualization is a spider chart summarizing the level of each practice examined by the tool. Developing the digital tool also required an iterative process that benefited from feedback from test respondents. Once the digital tool was finalized, the team conducted informal utilization tests with several audiences, including academics and non-profit organizations, and formal tests of validity that compared expert rankings of programs’ evidence base with self-assessments. The team used feedback from these exercises to further refine the tool.

The Way Forward
ENJOY has garnered interest in several dissemination events, with at least three organizations currently using the tool. Next steps will include identifying opportunities to strengthen the tool to adjust it to meet the needs of diverse organizations and to generate value by aggregating, analyzing, and sharing data collected by the tool, as its user base grows.

Author

• Alejandra Aponte, Mathematica