Context and comparability in ILSA: Illustrations of advanced methods

Group Submission Type: Formal Panel Session

Proposal

Context and comparability in ILSA: illustrations of advance methods.

International large-scale assessment (ILSA) studies provide a wide range and scope of educational achievement including academic outcomes such as math, reading and science. These different outcomes have been the traditional focus of many studies using ILSA data. Moreover, there is also a growth in these studies to include less traditional academic outcomes such as civic knowledge, 21st century skills and socio emotional skills. In this scenario, where more constructs are at stake, we can find more applications of inquiry onto different variables present in ILSA studies, besides academic achievement.

Researchers from different fields are augmenting their research tools for inquiry from population average models (McNeish et al., 2016), to different models from the generalized latent variable modelling framework onto complex sample survey studies (Stapleton, 2013; Sterba, 2009). Consequently, the research questions researchers are addressing with these additional tools of inquiry are varying. Population average models lead researchers to ask how different factors conditions the means of outcomes, while different latent variable models help to bring-in other research questions. This latter framework includes questions of comparability across participating countries, re-articulation of measures into newer views onto constructs, and newer proposals regarding how different factors are related to each other. In essence there is expansion of tools for research in large scale assessment for comparing countries, and contexts in novel ways.

The use of newer and more complex models to inquiry ILSA studies must deal with the parsimony/complexity tension. Some authors advocate that the likelihood of a more complex being true, with more parameters, is far-fetched (Saylors & Trafimow, 2021). Likewise, this position is contested arguing that, even if the simplest model of all, a pair of correlation, are more likely to be true than models with simultaneous equations, we need more complex models to address certain substantive issues, otherwise these would be left unaddressed (Del Giudice, 2021). The present panel favors the use of more complex models, under the conditions that these newer and more complex tools can shed light to relevant and substantive research questions.

In the present panel got together different authors, that employed more complex models and techniques to inquiry problems of comparability between measures to later, inquiry results across countries. Likewise, the present panel includes authors that have rethink the original measures included in ILSA studies, to addressed relevant and substantive questions for each of their fields.

In four presentations the panel authors illustrate the applications of different techniques from the latent variable model, to address comparability across countries, and enable contextual studies across populations and settings. Each author addresses the applicability of the chosen model to their research questions and describe the steps they have implemented to produce the presented results. Two presentations delve on issues of civic education using the International Civic and Citizenship Education Study. The first addressing political participation among students, using a conceptually driven model to represent many different repertories of political participation. The second, inquiries contextual effect of ideological climates in classroom groups on sexism. The two other presentations make use of Teaching and Learning International Survey. The first presentation relies on the alignment optimization method to inquiry distributed leadership, while the second uses a mixture model to study teacher motivation as different profiles.

Advantages

The present panel aims to share with the audience new ways to address research inquiries with ILSA data, including comparability problems in measurement terms, as well as comparability research problems in substantive terms. The present session is not intended as a workshop or standalone teaching material, but as an introduction to the applications of the methods here presented. The panel authors will explain the techniques they are using, and how these match their research questions, so to inspire researcher in the audience to explore how the presented methods of inquiry may fit their own research questions. The panel session will include a short discussion followed by a question-and-answer component.

The present panel assumes participant have only basic familiarity with educational measurement and psychometrics, thus the use of examples, relevant sources, and necessary background are a present emphasis in their presentations. The aim is presentations can be followed by non-experts, and only substantive interest is necessary to participate in the present panel. We are especially interested in social science researchers, educationalist and policy researchers interested in large scale assessment with no formal training in measurement or psychometrics, especially on the topics of civic education and teachers’ research.

Finally, this session is also of interest to researchers in the general field of large-scale assessment studies. It illustrates the application of different advance methods on ILSA studies, providing examples, tips and tricks, and code for enthusiast researchers who are aiming to explore and implement these different methods on their research. The aim is to share the application of these different methods in accessible manner to a general audience, while also providing tools for researchers who are in the journey of learning about these tools for their research.

Practicalities

Each author will be given 20 minutes to present their work. In each of these presentations, the authors provides and introduction to their work, and its research questions. That is, a summary of the topic, a description of the problem their work is addressing and their research proposal.

The emphasis of the presentation is on the research methods, their adequacy to answer their questions and the tools authors used to prepare their results. Additionally, authors are invited to address the challenges they faced in the application, tips and tricks in their approach, software use, and any advice they can brin to potential new users of the presented approaches. Finally, authors make their code available, so participants in the audience can make use of these to aid the adoption of new techniques.

In the next section, we include a summary of the four presentations of the panel.

Sub Unit

• Large-Scale Cross-National Studies in Education SIG

Chair

• Ernesto Treviño, Pontificia Universidad Catolica de Chile

Individual Presentations

• Cross-National Equivalence of Political Participation in Early Adolescence across 24 Countries. - Daniel Miranda, Pontificia Universidad Catolica de Chile

• Distributed Leadership and Alignment Optimization: A comparative, cross-cultural perspective across 40 countries. - Nurullah Eryilmaz, The University of Bath; Andres Sandoval-Hernandez, University of Bath

• Motivation profiles among teachers across 47 different countries. - Rosario Escribano, Centro de Justicia Educacional, Pontificia Universidad Catolica de Chile; Ernesto Treviño, Pontificia Universidad Catolica de Chile

• The Authoritarianism of Classroom Peers over Sexist Attitudes. A Good Case to Evidence Contextual Effects' Potential - Natalia Verónica López-Hornickel, University of Bath; Ernesto Treviño, Pontificia Universidad Catolica de Chile

Discussant

• Diego Carrasco, Centro de Medición MIDE UC, Pontificia Universidad Católica de Chile