Investing in Research and Development to Foster Innovation in PISA

Session Type: Coordinated Paper Session

Abstract

This session presents a collection of studies undertaken as part of the PISA Research, Development and Innovation (RDI) initiative, through which the PISA Governing Board aims to introduce conceptual, methodological and operational advances in how the OECD’s PISA study is conducted. PISA faces a number of challenges to the adoption of frontier technologies and of new methods, chief among them the large number of languages (about 100) and the wide variation in national contexts in which the assessment is conducted. These challenges are shared with other international large-scale assessments, such as the IEA’s TIMSS or PIRLS. All presentations in this session illustrate how these challenges play out throughout the assessment cycle, but also how they give rise to fertile research questions.
The session will in particular present results in three areas:
• the use of artificial intelligence and natural-language processing technologies for the automatic coding of open responses (2 presentations);
• the measurement of students’ socio-economic status;
• the use of adaptive technologies to increase accessibility in an international context;

Sub Unit

• Individual Papers, Innovation Demonstrations, Coordinated Sessions, or Organized Discussions

Session Organizer

• Francesco Avvisati, OECD

Individual Presentations

• Improving the Measure of Socio-Economic Status: Lessons from the PISA 2022 Cycle - Francesco Avvisati, OECD; Celine Wuyts, OECD

• Improving Accessibility in PISA for Students with Special Needs - Ava Guez, OECD; Ketan ., University of Massachusetts; Mario Piacentini, OECD

• Beyond Human Raters: Using AI and NLP in PISA’s Creative Thinking Tests - Ricardo Primi, University of Sao Francisco; Roger E. Beaty, Penn State University; John D. Patterson, Penn State University; Mathias Benedek, University of Graz; Ivandre Paraboni, Universidade de São Paulo; Denis Dumas, University of Georgia; Peter Organisciak, University of Denver; Tiago A Caliço, Organization for Economic Cooperation and Development; Mario Piacentini, OECD

• From Free-Form Job Descriptions to Standardized Occupational Codes: A Machine-Learning Approach - Tiago A Caliço, Organization for Economic Cooperation and Development; Francesco Avvisati, OECD

Discussant

• Leslie Rutkowski, INDIANA UNIVERSITY