Implementing Policy Linking in a Decentralized Education System

• In Event: Progress on Measuring SDG 4 – Improved Methodology, Community of Practice, and New Results for Policy Linking on SDG Indicator 4.1.1.

Proposal

The preparations and implementation of policy linking workshops for a decentralized education system followed the Policy Linking Toolkit (Authors, 2020). The policy linking workshops occurred first at the provincial level using a sub-national assessment and then later at the national level with a national assessment. It is expected that the process will continue in other provinces.
Preparations. The primary preparations common to each set of workshops entailed the adaptation of an agenda, facilitation slides, panelist background form, alignment rating form, item rating form, and workshop evaluation form. In both cases, the same local non-governmental organization provided the logistics and arrangements for the workshops. Analysts reviewed the assessments’ item-level classical test theory statistics, computed score distributions for different test forms, and created test characteristic curves (TCCs) in advance of the workshops.
The province’s sub-national assessment covered reading in two languages and math for the same grade-level. The provincial government provided substantial support in terms of personnel to provide in-person guidance on the analysis. The sub-national assessment had acceptable reliability in each subject; the item statistics revealed minor issues for item difficulty and discrimination. The TCCS were similar for different test forms.
In contrast, the national assessment had reading in one language at a middle primary grade-level; there was also math and reading in two languages at the end of primary. The reliabilities on each subject were acceptable; the item statistics showed a proper spread of item difficulties and discrimination was not problematic for any item. However, the TCCs varied for the test forms by subject, which showed that some forms were substantially easier or more difficult than others. This required postponement of the workshop until test equating could be conducted.
Implementation. The policy linking workshops followed a widely accepted benchmarking process (Cizek & Bunch, 2007). It used three tasks: 1. Checking the alignment of the assessment and the GPF, 2. Matching the items with the GPLs and GPDs in the GPF, and 3. Setting the benchmarks using a modified Angoff method (Angoff, 1971; Frisbie, 2003).
For the province’s sub-national assessment, the alignment process demonstrated that the reading assessments were minimally aligned, and math was additionally aligned. The panelists’ benchmarks between the two rounds did not change much for one language of reading, but the benchmarks from round 1 to 2 became higher (i.e., more difficult) on one reading language and math.
The workshop for the national assessment has not occurred. Nonetheless, the divergent experiences in the preparation process indicate that the workshop on the national assessment could be a very different experience from the sub-national assessment.
Conclusion. There was a high level of interest on the part of the provincial assessment offices. The federal office is undergoing a transition, with the goal of improving their technical and management capacity. Currently, there is more strength in the provinces than at the federal level, which is reflected in the quality of the assessments and the support for policy linking.

Author

• Sean Kelly, American Institutes for Research