Session Submission Summary
Share...

Direct link:

Policy Linking for Setting Global Benchmarks on Student Assessments with USAID-Funded Projects: Background, Pilots, Capacity Building, and Additional Benefits

Thu, April 29, 11:45am to 1:15pm PDT (11:45am to 1:15pm PDT), Zoom Room, 107

Group Submission Type: Formal Panel Session

Proposal

The 2017 Reinforcing Education Accountability in Development (READ) Act and the 2018 U.S. Agency for International Development (USAID) Education Policy specify indicators for reporting on student learning outcomes. These indicators are required for countries with USAID-supported basic education programs:

- Percent of learners targeted for USG assistance who attain minimum proficiency in reading and math at the end of grade 2 and the end of primary school;
- Percent of learners with a disability targeted for USG assistance who attain minimum proficiency in reading at the end of grade 2; and
- Percent of learners targeted for USG assistance who improve at least one proficiency level in reading and math at the end of grade 2 and the end of primary school.

Moreover, USAID aligned its indicators with Sustainable Development Goal (SDG) Indicator 4.1.1:

- Proportion of students a) in grades 2/3, b) at end of primary, and c) at end of lower secondary who meet at least a minimum proficiency level in reading and math, by sex.

There are two issues for USAID (and SDG) reporting on these indicators: 1) collecting assessment data in each country in the specified grades and subjects and 2) adjusting the difficulty level of the assessments from each country by linking them to a common scale. Solving the second issue – for fairness in global reporting – is the topic of this panel. Additional benefits from policy linking beyond global reporting will also be discussed.

Briefly, each country primarily (or exclusively) uses their own student assessments to measure student learning outcomes, rather than international assessments. Across countries, their assessment tools usually have some similarities in design but with different items and languages, i.e., with some common content but different levels of difficulty. For global reporting, having some common content is a requirement; however, different levels of difficulty inhibit reporting results in a fair way compared to other countries, unless the difficulties of each assessment are adjusted by linking them to a common scale.

There are several methods for linking different assessment to a common scale, both statistical and non-statistical. Statistical methods, such as test equating, require either including common items on different assessments or selecting common persons to take different assessments. This is not feasible on a global basis due to issues such as cost, security, timing, logistics, and language. Non-statistical methods, such as policy linking, require expert judgments about assessment difficulty in relation to a common scale. This is feasible on a global basis through standardized workshops in each country.

Policy linking is being developed through support from donor agencies – USAID, UNESCO, World Bank, FCDO, and Gates – as a psychometrically acceptable, practical, generalizable, and sustainable method to link assessments for global reporting. During its development, policy linking materials, namely the Global Proficiency Framework (GPF), have had additional benefits, such as providing a foundation for revising content frameworks, curricula, and assessments.

The GPF includes the common scale, with four levels of minimum proficiency with descriptions for grades 1 to 9 in reading and math. Assessments are linked to the common scale through global benchmarks, which reflect the difficulties of the different assessments. As an example, assume that National Assessment X (NAX) and National Assessment Y (NAY) each have scales of 0-100 points but different levels of difficulty. After expert judgments to link the assessments to the common scale, NAX, which is less difficult, has a “meets” global minimum proficiency benchmark of 60 points while NAY, which is more difficult, has a “meets” benchmark of 40 points. Students with the same ability would score 60 points on NAX and 40 points on NAY. After calculating the percent of students in the levels, the assessment results can be compared, aggregated, and tracked on a fair basis for global reporting.

Policy linking in each country involves three tasks in a five-day workshop with local experts (teachers and curriculum specialists—called panelists) and facilitators. In Task 1, the panelists use a standardized method to check the common content (alignment) of their assessment(s) with the GPF using the Frisbie (2003) method (2003). In Task 2, they match the assessment items with the minimum proficiency descriptions in the GPF. In Task 3, they set global benchmarks on their assessments using the Angoff (1971) method and the GPF.

By mid-2021, large-scale policy linking pilot studies will have taken place to 1) set global benchmarks in Bangladesh, Djibouti, Ghana, India, Kenya, Nigeria, and Senegal; and 2) define or revise minimum proficiency frameworks in Djibouti, Lebanon, Madagascar, Morocco, and Senegal.

There are four presentations in this session. They will 1) summarize the background and recent progress on developing the policy linking method; 2) explain how the GPF was used to define reading proficiency in Djibouti and Madagascar; 3) report on policy linking pilot studies for setting global benchmarks in Nigeria and Kenya; and 4) describe the training and next steps in capacity building for the policy linking method. The session will provide insights into policy linking for global reporting along with additional benefits. 

Sub Unit

Chair

Individual Presentations

Discussant