Using statistical linking to equate regional assessments results with international assessments– the Rosetta Stone initiative

• In Event: Highlighted Session: Reporting on SDG 4.1.1: A critical examination of the costs and benefits of emerging options

Proposal

International large-scale assessments (ILSAs) in education have been considered by many the most suitable source of information to measure and monitor progress towards several of the SDG 4 indicators. ILSAs offer systematic information from more than 100 education systems, provide information about the level of literacy and numeracy of children and youth, and have unrivaled data quality assurance mechanisms. Moreover, ILSAs produce data that is strictly comparable across countries and over time.

On the other hand, there are several regional assessments. For example, there is UNESCO’s Regional Comparative and Explanatory Study (ERCE; Estudio Regional Comparativo y Explicativo) in Latin America and Caribbean countries or the Programme for the Analysis of Education Systems (PASEC; Programme d’Analyse des Systèmes Éducatifs) in francophone sub-Saharan African countries. Regional studies deliver excellent information on achievement levels in the different regions. However, they are not linked to an international metric.

The International Association for the Evaluation of Educational Achievement (IEA) and the TIMSS & PIRLS International Study Center, Lynch School of Education at Boston College in cooperation with the regional coordination centers of PASEC and ERCE under the leadership of the UNESCO Institute for Statistics (UIS) and with support of the Gates Foundation has conducted a study to create a concordance table between the regional studies and IEA’s TIMSS and PIRLS, called Rosetta Stone. This approach is making use of all these invaluable sources of information. Students in selected countries who participate in the regional assessments also received a linkage instrument with test materials from TIMSS and PIRLS, allowing to measure students’ abilities by both the regional assessments and using the TIMSS and PIRLS metric. This enabled a sound statistical link between the regional studies and the ILSAs TIMSS and PIRLS. Evaluating latent correlations, we could see how well the study frameworks align and how comparable results are. This linkage approach is the most reliable approach to “translate” results from one study to the metric of another study.

As a result, concordance tables were established that match results from ERCE, PASEC and TIMSS, and PIRLS. Results from the regional assessments generate projections on the TIMSS numeracy and the PIRLS literacy scales. With this, also projections on the percentage of the students reaching international benchmarks – which are used to report the percentage of students reaching the minimum proficiency levels in literacy and numeracy (SDG 4.1.1) – could be established. The results from this project enable manyfold types of further analysis. More countries and more regional assessments participating in a follow-up of the Rosetta Stone study would increase the accuracy and quality of the projections for the regional assessments that were already part of the Rosetta Stone project. Also, further regional assessments could join the project to be mapped to the international TIMSS and PIRLS metric and other regional assessments already being part of Rosetta Stone.

Author

• Dirk Hastedt, IEA