Exploring Relationship between ICT Literacy and Global Competence through a Machine Learning Approach: Evidence from PISA 2018

• In Event: Educational assessments: insights from using various digital and quantitative approaches

Proposal

Background
Information and communication technology (ICT) has become a key component of teaching and learning in the digital era. ICT has transformed how people learn, communicate, and interact, and enabled collaboration that transcends geographic boundaries and cultural differences. By effectively integrating technology in the classroom, educators can create a more personalized, engaging, and sustainable learning environment (Romeo, 2001; Carrión-Martínez et al., 2020). In particular, virtual communication technology significantly enhances students’ cultural awareness, intercultural communication, and problem-solving skills by providing accessible and cost-effective opportunities for international collaboration (Li, 2013). During the COVID-19 pandemic, ICT played an even more crucial role in maintaining the functionality of education systems around the world, as schools in lockdowns rapidly adopted innovative educational technologies and transitioned to digital learning (Sá & Serpa, 2020; Ivaniuk & Ovcharuk, 2020; Antón-Sancho & Sánchez-Calvo, 2022). Additionally, in the post-COVID-19 era, the global educational landscape has been reshaped by the widespread of innovative learning technologies. Therefore, to be prepared for an increasingly interconnected world, it is essential for students to have sufficient ICT literacy, in other words, the ability to use digital technology, communication tools, and networks.
In its 2016 report, the United Nations Educational, Scientific and Cultural Organization (UNESCO)’s included the “global citizenship” education (UNESCO, 2016) as a core component of the Sustainable Development Goal 4.7. The pandemic made it even more important for students to be able to address global challenges and adapt to a rapidly changing labor market. With this being said, the education of the post-COVID era must focus on developing students’ global competencies, such as international competitiveness, intercultural communication, collaboration skills, and adaptability to diverse cultural contexts and economic changes (Caligiuri & Di Santo, 2001; NEA, 2010). In response to such a need, in 2018, the Program for International Student Assessment (PISA), an international large-scale assessment that traditionally measures 15-year-old students’ reading, mathematics, and science literacy, administered its Global Competence (GC) assessment for the first time.
Research Questions
Given the growing importance and interest in ICT literacy and global competencies, this study examines the empirical evidence from PISA 2018 and investigates the relationship between students’ ICT literacy and their global competencies through applying a machine learning approach, namely random forest (RF). Specifically, this research explores how students’ ICT literacy and ICT-related experiences inside and/or outside of school contribute to their performance in the GC assessment. The following three research questions (RQs) will be addressed.
1. How do ICT-related variables predict students’ global competence achievement in PISA 2018?
2. Which ICT-related variables are the most significant predictors of students’ global competence?
3. How does the relationship between ICT-related variables and GC assessment scores vary across different student characteristics, such as gender, family socioeconomic status, and school type?
Data and Methodology
To answer the above RQs, we will analyze empirical data from the most recent cycle of PISA administration. The PISA 2018 GC assessment adopted a multi-method, multi-perspective approach that contains two instruments: the cognitive test assesses students’ knowledge and skills related to global and multicultural issues, and the student questionnaire focuses on both students’ cognitive and socio-emotional skills and their attitudes toward global issues (OECD, 2020). In this study, we will use the students’ cognitive test scores of the GC assessment as the outcome variable. Based on their GC assessment scores and PISA’s definition, we will assign students into “high”, “average”, and “low” performance groups. PISA 2018 Insights and Interpretations (OECD, 2019) document defines students who scored in the top quarter of each assessment as high achievers. Students who scored in the bottom quarter will be classified as low achievers, and the rest of the students will be labeled as average achievers.
From the PISA 2018 data, we also selected two additional sets of variables as predictors in the analyses: ICT-related variables and students’ demographic variables. Specifically, seven ICT-related indices are obtained from the PISA student questionnaire, such as perceived ICT competence (COMPICT) and ICT use outside of school for leisure (ENTUSE). Students’ demographic characteristic variables include gender, family socioeconomic status, highest parental education level, and school type as another set of the independent variables.
Since most of the variables in PISA data are quantitative, researchers typically utilize statistical techniques to analyze PISA data. In this study, we will apply a machine learning method, namely random forest (RF), to answer our RQs. RF is a commonly used and powerful method for classification tasks. It is chosen for analyses of this study because of its high classification accuracy, strong interpretability, and effectiveness in processing the complexity and variance in datasets (Qiao and Jiao, 2018), such as the PISA dataset. With RF, it is relatively straightforward to understand and identify the variables, among many of them, that impact students’ GC assessment scores.
Research Significance
In this study, we aim to discern how ICT-related variables from the individual student questionnaire, alongside student characteristics, predict each student’s GC scores. Findings of this study will provide more insights into how education systems can better integrate ICT to foster the development of globally competent individuals. By uncovering hidden or unexpected patterns and relationships in the data, we hope to identify predictive variables that will be of interest for researchers to follow up on with additional studies. We might also discover findings that lead us to question the construct validity of the GC score, which would align with some scholars who have expressed skepticism about this measure (Chandir & Gorur, 2021; Bailey et al., 2022).

Authors

• Suliya Nijiati, Indiana University - Bloomington

• Sijia Huang, Indiana University Bloomington (IUB) - Bloomington, IN