BackHome
Refresh: OffView Personal Schedule
Session Submission Summary
Share...

Direct link:

Protesting hegemonic definitions of quality education evidence: The case for a wider range of evidence

Wed, March 13, 4:45 to 6:15pm, Hyatt Regency Miami, Floor: Terrace Level, Jazmine

Group Submission Type: Refereed Roundtable Session

Proposal

There is an increasing desire in the global education sector to build a stronger culture of using evidence to inform decision-making in both policy and practice. Dominant definitions of “good” or “quality” evidence in the education sector are usually limited to research published through academic and traditional channels and display high methodological rigor. Such sources are often inaccessible to educational decision-makers due to paywalls and language that is difficult to understand and interpret. From a decision-maker's perspective, quality evidence speaks to the kinds of decisions they must make, comes from a trustworthy source (most often government itself), is contextually rich, is accessible in multiple senses, and captures stakeholder perspectives. (Education.org 2021; OECD 2022). This large volume of evidence, the so called “grey literature,” is undervalued. How should we define quality when looking at the extensive amount of locally-contextualized and practice-derived evidence produced by non-governmental and civil society organisations, and researchers based in the Global South?

While there is immense value in widening the education evidence, there are specific barriers to increasing its use in education policy and practice. The first is the difficulty to identify and access it. A second is the lack of a typology or classification system for this diverse universe of evidence sources. But the most contentious barrier is the issue of appraising the quality of grey literature evidence because standard appraisal methods focus heavily on methodological rigor to determine quality. This refereed roundtable presentation will focus on this third challenge around the definition of quality of evidence and how current definitions limit the voices and accessibility of evidence in education. The presenters in this roundtable are all members of an International Working Group designed to help expand the types of evidence, and therefore voices, available to decision-makers beyond those published through academic and commercial channels. It aims to build on existing appraisal guidance and develop a new, coherent, and intellectually rigorous system to make locally generated evidence and grey literature easier to use in educational decision-making.

The issue of quality of grey literature is important because not just any piece of writing should be used as evidence. However, significant amounts of grey literature in education are not research products, so the well-established criterion for appraising traditionally published research is not fully appropriate. Building on the initial deliberations and debates of the International Working Group at its initial convening in June 2023, this Roundtable continues and deepens the discussion on defining quality for grey literature.

In looking at the issues of quality in education evidence, this Roundtable will address the questions of:
1. What is the equity argument for including grey literature? Whose voices are we eliminating if we maintain the standard definitions of quality?
2. How does widening the range of evidence enrich the quality of evidence in the sector?
3. Who sets the standards of evidence quality? Creators or users?
4. How does a wider definition of quality relate to rigor?
5. Beyond methods, what additional metrics could help in defining evidence quality for grey literature?

Working to use a broader range of evidence in education decision-making would be a major methodological contribution to the education sector. In a recent landscaping exercise, Education.org studied 26 initiatives to identify and learn from existing practices in synthesising education evidence. The landscaping found that, despite its great value to decision-makers, grey literature is not widely considered evidence. Only 9 of the 26 initiatives include grey literature in their definition of evidence and generally it must follow rigorous methodological protocols that exclude large amounts of unpublished evidence. Additionally, rarely was an explicit concern for the diversity of stakeholders represented in the evidence. When beneficiaries are mentioned as part of the evidence review, their roles are limited to involvement in prioritisation, design, implementation, and dissemination, not as experts themselves offering evidence.

Widening the evidence to include grey literature would have long-term impacts on education to diversify the voices in evidence and prioritize local and contextually rich evidence. Including more practice-derived and contextualized evidence would promote a shift in global knowledge flow for local evidence playing a bigger role in informing local and global contexts. Additionally, widening the evidence base would influence the types of evidence funded and generated to better serve decision-makers. Resulting in more accessible and usable evidence in decisions.

Maintaining hegemonic definitions of quality of evidence severely limits both the types of evidence available and the voices included in and considered as evidence. In line with the 2024 CIES conference theme of “The Power of Protest,” this roundtable seeks to be a constructive disruptor of the status quo in defining evidence quality and be more inclusive of the voices and perspectives represented in evidence. The intention is to be additive to existing appraisal protocols and ensure that a wider range of voices are brought to the table to influence education decisions. Expanding the definition of quality will help incorporate more diverse local voices and deepen the culture of evidence use in education around the world.

Sub Unit

Chairs

Individual Presentations

Discussants