Creative Thinking in PISA 2021: Perspectives on Designing and Developing an Innovative Domain

• In Event: Thinking Globally, Acting Globally: Research and Assessment Collaboration and Capacity-Building in International Settings

Abstract

The Programme for International Student Assessment (PISA) is a multi-faceted, highly complex triennial survey designed to collecting evidence of student achievement among 15-years-olds from countries all over the world. Since its initial administration in 2000, the areas of reading, mathematics and science have made up the core of its cognitive tests, alongside a background questionnaire aimed at noncognitive traits and other contextual factors. Taken together, the steady bulk of PISA is “designed to gauge how well the students master key subjects in order to be prepared for real-life situations in the adult world” (https://www.oecd.org/pisa/). One special feature of PISA, however, is the introduction of a new construct that varies in each 3-year cycle, and that is the Innovative Domain.
As part of its regular effort build an international contractor consortium in an advance of each forthcoming cycle, the Office for Economic and Cooperative Development (OECD) holds an open competition for the major administrative and operational aspects of this endeavor. And so it was in 2017 that ACT, Inc., known for its flagship college and career readiness assessment, took its national know-how to the international testing arena by successfully bidding on a contract to design and develop a measure of creative thinking—the innovative domain for PISA 2021.
By the time ACT was notified of our winning proposal, the OECD was already convening a group of creative thinking experts to define the construct and settle on a competency model. ACT inherited responsibility for this group and their work, and continued to advance the instrument development using an evidence-centered design approach (Mislevy, Steinberg and Almond, 2003).
At its core, ECD provides a strong foundation for the development of a valid assessment of the complex and multidimensional construct of creative thinking. It requires documented, explicit linkages among the test purpose, the claims made about test takers, the evidence supporting the claims, and the test takers’ responses to the tasks providing the evidence. Adopting the ECD process in the context of the PISA 2021 necessitated following of a rigorous sequence of steps, as well as some creative thinking given the constraints of such a standardized assessment with such diverse conditions and populations.
In this presentation, I reflect on both the research and development challenges that ACT—alongside its fellow international contractors—had to navigate to build a valid and reliable assessment of creative thinking. I will also share some of the key insights gained throughout the intricately collaborative work that went into the building of the innovative domain for PISA 2021.

Author

• Ryan OConnor, International Baccalaureate