Linking the second and third round of UNESCO’s Regional Comparative and Explanatory Study

• In Event: Comparative Studies in Education (I)

Proposal

UNESCO’s Regional Comparative and Explanatory Study (ERCE) is the largest large scale assessment in Latin America. It assess 3rd and 6th grade students in the subjects of Reading, Mathematics and Science (UNESCO OREALC, 2021). Since its beginning, in 1996, there have been four rounds of ERCE. Unfortunately, it is only since the third round (TERCE) that the results are located in the same scale. That means that, even though there are up to 4 measurement points for participating countries, only the last two of them can be actually used as a measure of trends. In the UNESCO repository it is possible to find trend results between the second and third round of ERCE (UNESCO OREALC, 2014), but these are based on data processing that allowed the generation of TERCE results on the scale of the second round, SERCE (UNESCO OREALC, 2016). In other words, there are two versions of TERCE results, one that is comparable to SERCE and another that set the scale that UNESCO currently uses for its studies.

This is not a unique situation in ILSAs. IEA's TIMSS had a first phase with two mathematics studies, the First International Mathematics Study (FIMS) and the Second International Mathematics Study (SIMS), in addition to their respective First and Second International Science Study (FISS and SISS, respectively) (Guhn, et. al. 2014). It is from its second phase, with the Third International Mathematics and Science Study (TIMSS), later renaming Third to Trends, that the IEA sets the reporting scale that it currently uses. This has left trend analysis with the first two studies out of the official results (Majoros, 2023).

Some efforts have been made to recover the useful information provided by these two initial studies, establish a link to the second phase studies, and thus obtain a longer timeline for analyzing trends in educational attainment (Afrassa, 2005; Majoros et. al., 2021 ; Majoros, 2023).

Following those studies, this work aims to put the data of ERCE’s second round, SERCE, in the current results scale. This will allow data users to use and analyze the data of the second round in the same scale as in the third in fourth round and with the same methodologies. This is even more relevant in Latin America due to the lack of other sources of long-standing information, and with the level of coverage that UNESCO studies have, which allow analysis of trends in the region's educational systems. For this purpose, there are several conditions and steps that should be followed.

In first place, the raw data of SERCE has to be processed with the same data cleaning criteria used in the third and fourth round. Then, cleaned data has to be modelled with the currently used specifications. This includes the use of a Rasch model, the use of the same criteria for item selection, drawing of plausible values and the transformation of the scores to the ERCE scale.

It is also important to pay attention to the methodology that will allow to link these results to the current score scale. For this task, we will take advantage of the booklets design of the third round, which included a set of items of SERCE (UNESCO OREALC, 2016). These items will be used to link the results between these two rounds.

Regarding the assessment framework, since UNESCO's studies are curricular-based, there is detailed information on the procedure for establishing this framework. A minimal adjustment to the set of items is expected to be made to maintain those that have a higher level of comparability in the measured construct.

Finally, an adjustment will be made to the sample weights in order to correctly expand the results to the target population.

So far, some advances have been made regarding data processing and initial definitions on the fit of measurement models. For this presentation, it is expected to show the results obtained from the linking between the second and third round of ERCE with some preliminary results of trends on the populations assessed.

References

Afrassa, T. M. (2005). Monitoring mathematics achievement over time: A secondary analysis of FIMS, SIMS and TIMS: A Rasch analysis. In S. Alagumalai, D. D. Curtis, & N. Hungi (Eds.), Applied Rasch measurement: A book of exemplars. Papers in honour of John P. Keeves (pp. 61–77). Springer.

Guhn, M., Gadermann, A., Wu, A.D. (2014). Trends in International Mathematics and Science Study (TIMSS). In: Michalos, A.C. (eds) Encyclopedia of Quality of Life and Well-Being Research. Springer, Dordrecht. https://doi.org/10.1007/978-94-007-0753-5_3063

Majoros, E., Rosén, M., Johansson, S., & Gustafsson, J.-E. (2021). Measures of long-term trends in mathematics: Linking large-scale assessments over 50 years. Educational Assessment, Evaluation and Accountability, 33(1), 71–103. https://doi.org/10.1007/s11092-021-09353-z

Majoros, E. (2023) Linking the first- and second-phase IEA studies on mathematics and science. Large-scale Assessment in Education 11, 14. https://doi.org/10.1186/s40536-023-00162-y

UNESCO OREALC (2014). Comparación de resultados del Segundo y Tercer Estudio comparativo y explicativo: SERCE y TERCE, 2006-2013. https://unesdoc.unesco.org/ark:/48223/pf0000244239

UNESCO OREALC (2016). Tercer Estudio Regional Comparativo y Explicativo: Reporte Técnico. https://unesdoc.unesco.org/ark:/48223/pf0000247123

UNESCO OREALC (2021). Los aprendizajes fundamentales en América Latina y el Caribe. Estudio Regional Comparativo y Explicativo (ERCE 2019). https://unesdoc.unesco.org/ark:/48223/pf0000380257

Authors

• Carlos Cayumán, Universidad de Chile

• Fabián Fuentealba, UNESCO

• Francisco Gatica, UNESCO