Profiling Behavioral Patterns in Interactive Computational Thinking Tasks Across Countries: Sequence Mining With Process Data

• In Event: Assessment in the Advent of Process Data and AI

Abstract

Computational thinking and adaptive problem-solving skills in digital competence tasks consist of the abilities of individuals to use computers to collect, manage, produce, and exchange information as well as formulate solutions to problems. These have been recognized across countries as among the most important skills in the 21st century. The International Computer and Information Literacy Study (ICILS) in 2018 organized by the International Association for the Evaluation of Educational Achievement (IEA) extends the evaluation of students’ computer and information literacy (CIL) skills and introduces a novel assessment of students’ computational thinking (CT) skills, defined as the ability to recognize, analyze, and describe real-world problems so their solutions can be operationalized within programming tasks (Fraillon et al., 2019).

The aim of this study is two-fold: given limited availability of process data in fine-grained level, we focus on testlet level, namely, to extract behavioral patterns from the whole CT unit. First, to cluster the behavioral patterns into meaningful groups, thus extract representative time allocation patterns through the 9 items within one unit, and second to evaluate the CT skills by each latent group to identify the most optimal pattern across countries. Eligible process data recorded during students programming problem-solving process could provide a new angle to understand how students learn, interact, and adapt their computational thinking skills to solve digital tasks in the programming environment.

To achieve this goal, we conduct two sub studies: (1) We conducted a cluster analysis on timing and process-related variables, to group students with homogeneous patterns and map with students’ CT and CIL latent scores and background variables across countries. (2) We focus on command efficiency and time allocation pattern from process data to calculate the sequence distance and extract representative patterns when students solving coding task throughout the whole unit. This sub-study focuses on better understanding how students with missing responses allocate their time and pinpoint potential reasons for their missing in certain items.

In summary, this study provides a new angle on measurement of students’ CT and CIL skills with the utilization of process data. Advanced process data analysis either by variable-based approach or machine learning techniques or their combination are in pressing trend and worthwhile to have an extensive and intensive research in the near future especially in cross-country comparison.

Author

• Qiwei He, Georgetown University