Operationalizing Policy Linking – Recommendations and Next Steps

• In Event: Linking assessments to a global definition of minimum proficiency using policy linking

Proposal

The purpose of this presentation is to provide practical guidance and recommendations for operationalizing the policy linking method in linking assessments to a global standard, e.g., to a standard of minimum proficiency. The recommendations are presented in four categories below.
1. Develop a toolkit for piloting the policy linking approach: A toolkit can be developed for piloting purposes. It would provide a detailed justification and guidance for the steps in applying the policy linking approach to link existing assessments with the global standard. Discussions on the toolkit and technical paper would take place between technical stakeholders such as donor agencies and implementing partners.

2. Develop policy definitions and performance level descriptors (PLDs) for a global standard: In accordance with the guidelines in the toolkit, a policy definition of the global standard can be either adopted or adapted from an existing definition. A diverse group of SMEs would be selected, including language experts, teachers, curriculum developers, teacher trainers, and assessment experts. The SMEs would develop PLDs, e.g., for reading and math at lower and upper primary school, from global content frameworks, such as those extracted from assessments by IBE-UNESCO. The resulting global PLDs would need to be broad enough to cover a range of assessment content and formats as well as diverse rates of knowledge and skills acquisition.

3. Select assessments or parts of assessments (i.e., testlets or subtasks) for policy linking. In some cases, entire assessments can be used for policy linking while, in other cases, only parts are used. For instance, in early grade reading, there might be a recommendation to use either oral reading fluency (ORF) or reading comprehension, or a combination of the two, as a proxy for overall reading knowledge and skills. Reading experts can provide advice on making this choice. They may determine that ORF raises issues due to an over-influence of language on measuring minimum proficiency, which would in turn have effects on comparability and aggregation. The experts might say that comprehension is more appropriate as a subtask for this purpose.

4. Apply methods for setting and validating cut scores for policy linking: Assessment specialists and psychometricians can recommend internationally accepted and validated standard setting methods that would be applicable to different assessment formats. Trained psychometricians and SMEs can then apply the appropriate standard setting method to set cut scores on assessments (or subtasks), based on the PLDs. As a part of policy linking, validity evidence would be gathered to provide information on the process and outcomes to establish confidence in the cut scores.

After the toolkit is piloted and the links are validated, the toolkit can be revised and finalized for use with other assessments for linking to the global standard. The implementation of the policy linking method should be conducted using best practices. It should provide adequate validity evidence to permit accurate comparability and aggregation.

Authors

• Idalia Rodriguez-Morales, Management Systems International

• Dana Kelly, Management Systems International (MSI)