Mathematics Benchmarking using Policy Linking for SDG 4.1.1 Reporting

• In Event: New approaches for measuring and comparing numeracy education outcomes across countries

Proposal

Starting in 2017, under the guidance and funding of international stakeholders, a team of psychometricians and content experts adapted the policy linking method for reporting on Sustainable Development Goal (SDG) 4.1.1 and similar bilateral indicators involving student learning outcomes (Ferdous, Kelly, & Davis, 2019).
Policy linking has been cited in the assessment literature for the past 15-20 years (Cizek & Bunch, 2007; Lim, et al., 2013; Buckendahl & Foley, 2015). The main feature is using a judgmental method, such the Angoff method (Angoff, 1971), to set global benchmarks (or cut-scores) on different assessments in relation to a common content framework and measurement scale. Percentages of students by performance level are calculated by applying the benchmarks to operational assessment data, and descriptors from the content framework are used to provide meaning to the scores by level. Policy linking has been approved by the UN and other agencies for global reporting on learning outcomes.
SDG 4.1.1 requires reporting results on reading and mathematics assessments by each country at three grades: end of lower primary, end of upper primary, and end of lower secondary. The policy linking method was piloted for the two subjects at various grades in 12 countries from 2019-2021. The final method has been used operationally in several additional countries in 2022. Donor agencies supporting policy linking include the World Bank, USAID, FCDO, UNESCO, UNICEF, ACER, and the Gates Foundation.
This presentation focuses on mathematics benchmarking in two countries using the final policy linking method. For Country A, the assessment was group administered to a large sample of Grade 5 students in 2021, with benchmarking in early 2022. For Country B, the assessment was individually administered to a large sample of Grade 3 students in 2022, with benchmarking mid-2022. Neither assessment used timed tasks, though Country B administered some subtasks that have been used as timed tasks in other countries.
In the two countries, policy linking was implemented as follows: 1) training for government officials and 2) workshop involving teachers and pedagogy experts (“panelists”). The training took three days and the workshop five days. A workshop can accommodate two or three assessments (e.g., reading and/or mathematics at different grade levels), with about 15 panelists per assessment, along with facilitators.
Global benchmarks were successfully set on the assessments in both countries. The panelists followed the policy linking method involving alignment (Task 1), matching (Task 2), and benchmarking (Task 3) under the guidance of the facilitators. Evaluations from both workshops showed that the panelists understood the tasks and felt confident about their benchmarks.
Results include overall benchmarks (averages from the panelists), location statistics (benchmarks of the panelists in relation to those of other panelists), impact data (percentages of students in performance levels), standard errors of the benchmarks, and inter- and intra-rater reliability. The technical reports from the workshops included background, method, results, lessons learned, and recommendations. These results will be provided during the panel presentation.

Author

• Jeff Davis, American Institutes for Research