Revised Technical Guidance for Policy Linking

• In Event: Progress on Measuring SDG 4 – Improved Methodology, Community of Practice, and New Results for Policy Linking on SDG Indicator 4.1.1.

Proposal

The GPF and Policy Linking Toolkit (PLT) were created as global public goods to support country governments that want to report to SDG 4.1.1, but that do not buy into international or regional assessments. The tools provide a methodology by which countries can link their national and/or sub-national assessments to the GPF and SDG 4.1.1 as well as to one another. However, given the newness of the GPF and policy linking to the international development context, UIS and its partners facilitated two rounds of piloting of the tools with the goals of identifying how they work in different language contexts, with different types and lengths of assessments, with different facilitators, and virtually (as a result of COVID-19) versus in-person. The first round of pilots was completed in 2019 and led to the first revision of the PLT and the GPF – both of which were finalized in the spring of 2020. The second round of piloting wrapped up in 2022, and it led to a number of lessons that the international community agreed to incorporate into the PLT and its related materials. Further, the Bill & Melinda Gates Foundation also funded an evaluation of the policy linking process, which resulted in a report and recommendations on ways to improve the PLT. In 2022, UIS funded a partner to incorporate these changes into the PLT and its related documents.

UIS is pleased to present the revised PLT and related materials. In this paper and presentation, UIS will share the revised guidance on the quality of assessments that can be used for reporting on SDG 4.1.1 using policy linking and what changes have been made to the policy linking process and PLT. Specifically, they will discuss the:

• Tiered criteria for an assessment to be considered valid and reliable to allow linking to SDG 4.1.1 through a policy linking workshop, including details on assessment design, item reliability/validity/fairness, sample design, distribution/selection of booklets, and data cleaning. This section includes details on the minimum criteria for conducting a workshop for capacity-building purposes only versus criteria for conducting a workshop with the intention of reporting on SDG 4.1.1;
• Details for differences in considerations for an Item Response Theory-based assessment versus one based on Classical Test Theory;
• Guidelines on selecting an assessment booklet to link through policy linking when countries have multiple booklets (as well as steps to take to link the remaining booklets);
• Guidelines for dealing with missing data;
• Guidelines for dealing with translations of assessments;
• A step for determining text level/complexity for policy linking workshops that seek to link reading assessments;
• A requirement for countries to report on precision, accuracy, and consistency, using the formulas included in the existing PLT annexes;
• Guidelines on how to proceed if there are outlier ratings after Round 2 ratings in a policy linking workshop; and
• Guidelines on how the PL method works with constructed response items.

This paper is relevant for anyone wanting to use the GPF or PLT in the future.

Author

• Silvia Montoya, UNESCO Institute For Statistics