A critical cartography of large-scale international assessments

• In Event: Critical perspectives on large-scale assessments: Methodological innovations, validity, and societal implications

Proposal

The recent years have seen an increase in calls to collect empirical data in order to measure the health and performance of educational systems across the world. Many governmental decision-makers have accordingly invested in large-scale assessments in order to both compare and improve their educational systems, as well as to remain competitive on the international stage. Regular assessments are indeed critical to making informed policy decisions as they provide thorough measurements which go beyond student learning outcomes.

Previous research has been conducted on the challenges and best practices experienced by large-scale assessments (Ertl, Hartmann & Heine, 2020), such as issues of representativity and comparability across countries and time. However, there is currently little critical discussion around the theoretical foundations of assessments (Bodin, 2023), let alone whether the selected measurements are aligned to the 4th industrial revolution we are facing. This oral communication aims to fill this gap by conducting a critical cartography of large-scale international assessments.

The following metrics were included: geographical regions where data is collected, theoretical frameworks at the foundation of assessed domains, student population being surveyed, other respondents, processes of creation and adaptation of the assessments, sampling methods and recruitment, test content and administration, and funding procedures. Six large-scale assessments were included in our study: EGRA, PISA, TIMMS, PIRLS, PASEC and SACMEQ. Data was extracted from the technical reports published by the organizations responsible for each of the included assessments.

Inductive content analysis and map analysis revealed several spectra bounded by opposing concepts. For instance, the included large-scale assessments are all positioned differently on each spectra according to the importance they give to several concepts, such as:
- Antinomy between methodological standardization (e.g., PISA) and cultural adaptation (e.g., EGRA, PASEC);
- Antinomy between global learning outcomes attuned to international standards (e.g., TIMMS) and the pull towards national evaluations (e.g., EGRA);
- Infrastructure antinomy between increasing the number of surveyed countries and availability of internal resources to conduct the assessments;
- Subsequent longitudinal analyses also revealed an antinomy between increasing alignment of the tests to 21st century skills (including environmental awareness and ICT skills) and measuring foundational skills such as literacy and numeracy.

These findings question the purpose of large-scale international assessments as the dichotomy between global-local challenges their very essence of providing standardized measures of key student learning outcomes. This communication will shed the light on additional complementary purposes by first discussing how large-scale international assessments could help improve educational systems not only for students but also for the entire school community, and second, how they also provide useful national contextual information beyond international rankings (Brochu, 2020).

Authors

• Valerie Vinuesa, CONFEMEN

• Gabriela QUINTELA DO CARMO, Université de Sherbrooke

• HILAIRE GUY HOUNKPODOTE, CONFEMEN