Highlighted Session: Reporting on SDG 4.1.1: A critical examination of the costs and benefits of emerging options

Group Submission Type: Highlighted Paper Session

Proposal

Subtheme III of CIES 2023 challenges us to examine how we can improve school systems to better support and contribute meaningfully to education for all. This challenge is a central component of the Education 2030 Framework, which encourages education systems to promote inclusive and equitable access to quality education, ensuring “that all girls and boys complete free, equitable, and quality primary and secondary education leading to relevant and effective learning outcomes”. In monitoring progress towards this goal, the Framework asks countries to report against a series of indicators that cover the entire learning cycle. Indicator 4.1.1., for example, requires countries to determine the “Proportion of children and young people (a) in Grade 2 or 3; (b) at the end of primary education and (c) at the end of lower secondary education achieving at least a minimum proficiency level in (i) reading and (ii) mathematics, by sex”.

The adoption of indicator 4.1.1 challenges stakeholders to identify optimal approaches for measuring proficiency levels, and to do so in ways that allow for comparability across space (countries or subnational jurisdictions) and time. Measuring the quality of education systems at specific points in time allows decision-makers to determine if education systems are addressing the learning needs of all learners and providing them with minimal proficiencies required to develop into successful, engaged citizens. Comparing learning results over time allows these same decision makers to assess if the quality of their education systems is improving. These two data points provide country-level decision-makers with the insights necessary to drive policy reform and resource allocation.
Another equally important data point for national decision-makers is the ability to compare the quality of their education system with that of other countries in the same geographic region or economic group, or across the globe. Identifying whether learners at key grade levels are achieving the same minimal proficiency levels as their counterparts in other countries is essential in determining whether the quality of an education system aligns with a country’s political, social, and economic aspirations.

Initiatives to identify reliable and valid approaches for comparing learning results over time and space, both for national reporting and for reporting on SDG 4.1.1, have revealed the complexity of the issue (see Gustafsson, 2019). Technical comparability requires that countries, jurisdictions, or even regional or international testing organizations agree:
• On the key knowledge and skills that should be measured at each grade level, so that assessment instruments used are comparable in terms of breadth and depth of coverage. This is defined in the 2020 Global Proficiency Framework (GPF), a reference document for mapping the content of national assessments.
• On the definition of Minimal Proficiency Levels (MPL) in mathematics and reading concerning the key knowledge and skills measured. This is essential for determining the percentage of learners achieving this level of proficiency. The MPLs were developed to support SDG 4.1.1 reporting and agreed upon by the countries.
• On the methodological processes for comparing results across assessment instruments, contexts, and time.
• On the minimal procedures required to ensure the quality of the data collected and to validate findings.

This panel examines emerging options for addressing the last two elements. It examines the technical, political, and financial costs of three different avenues for reporting valid, comparable data on learning outcomes:
• A professional calibration or judgment method using national assessments (Policy Linking) and a Pairwise method, where national and international experts identify the benchmarks or assessment cuts scores that correspond to the MPLs or a given grade and domain (reading, mathematics).
• A statistical recalibration method (Rosetta Stone) that links the results of regional assessments with the TIMSS and PIRLS achievement scales, making it possible for participating countries to map their learners’ performance onto the TIMSS and PIRLS scales.
• The use of common testlet(s) pre-aligned to the MPLs (Assessments for Minimum Proficiency Levels or AMPLs).

How the contribution builds on existing work – The MPLs were developed to support reporting on SDG 4.1.1. Attention is now turning to how countries can use this common definition of minimal proficiency to ensure the validity and comparability of data over time, particularly concerning the attainment of proficiency benchmarks (see for example The Protocol for Reporting on the Global Indicator 4.1.1 (UNESCO, March 2022). Over the past two years, the three methodologies outlined above have has been piloted in several countries over the past two years. Each methodology offers different costs and benefits for governments. This presentation will introduce participants to three methodologies and present them with an analysis of budgetary, opportunity, human capital, efficiency, and equity costs and benefits of each methodology (see UNESCO Information Paper no. 53, January 2019).

The theoretical contributions situated in terms of linguistic, national, or geo-political contexts – The different methodological approaches explored have been developed and field tested in a variety of national and geo-political contexts, and the analysis of costs and benefits is informed by this.

Implications for future practice, policy, or theory- The comparative analysis of the costs and benefits of each methodology will allow participants to make informed decisions about the methodology best suited for a given context. At the same time, panel members spearheading each approach will receive feedback from participants on potential unexamined costs and benefits of their approach. The exchange should inform future policy decisions.

The originality of the proposal – Many countries or jurisdictions are beginning to identify potential avenues for reporting SDG 4.1.1. This proposal will present them with three different options and considerations to weigh when selecting the option that will best respond to their contextual needs and priorities.

Sub Unit

• Monitoring and Evaluation SIG

Chair

• Luis Crouch, RTI

Individual Presentations

• Using National Assessments data and the revised Policy Linking procedures to report on SDG 4.1.1 - Sanneke Schouwstra, CITO

• Using assessment items from the UIS and other item banks - the AMPL initiative - Maurice Walker

• Using statistical linking to equate regional assessments results with international assessments– the Rosetta Stone initiative - Dirk Hastedt, IEA

Discussant

• Ramya Vivekanandan, Global Partnership for Education