Analyzing Process Data From Problem-Solving Items With N-Grams: Insights From a Computer-Based Large-Scale Assessment

• In Event: Technology Tools for Real-World Skill Assessments in Schools

Abstract

Computer-based assessments (CBAs) provide new insights into behavioral processes related to task completion that cannot be easily observed using paper-based instruments (Goldhammer et al., 2013). This chapter draws on process data from log files recorded in a computer-based large-scale program, the Programme for International Assessment of Adult Competencies (PIAAC; cf. Schleicher, 2008), to address the question of how sequences of actions recorded in problem-solving tasks are related to task performance.
The purpose of this study is twofold: first, to extract and detect robust sequential action patterns that are associated with success or failure on a problem-solving item, and second, to compare the extracted sequence patterns among selected countries. We investigated the utility of behavioral process data for predicting differences in task performance in the PIAAC problem solving in technology-rich environments (PSTRE) domain. More specifically, we separated the database into two performance groups (correct and incorrect) in a sample item, extracted action sequences, and identified the key action sequences that were significantly associated with task completion.
A total of 3,926 test takers from three exemplary countries (the United States, the Netherlands, and Japan), consisting of 2,754 individuals (70.1%) in the correct group and 1,172 (29.9%) in the incorrect group, were included in the study. We focused on the sequence data resulting from the task requirements of one PSTRE item. This item consists of two environments: a spreadsheet environment that contains a database with the information required to solve the task, and an email environment to provide the response (OECD, 2009). Motivated by the methodologies of natural language processing and text mining, we utilized n-grams to disassemble the test takers’ process data into small action sequences (i.e., unigrams, bigrams and trigrams) and applied chi-square feature selection model (Oakes et al., 2001) to extract robust action features that facilitate differentiation between performance groups at a variety of aggregate levels.
The results showed that action sequence patterns significantly differed by performance groups and were consistent across countries. Among the robust indicators that we noticed were that the correct group had a better understanding of the subgoals of different environments and were more likely to recover from initial errors in the problem-solving process. Conversely, respondents in the incorrect group appeared to have only a relatively vague idea about what was expected in the item and were more likely to show hesitative behaviors, such as clicking on the cancel button multiple times and using the help function. This study also demonstrated that the process data were useful in detecting missing data and potential issues in item development.
In conclusion, with increasing use of computer-based assessments, process data play an increasingly important role in tracking test takers’ thinking and action sequences. This pilot study presented what we think is a promising method to analyze process data and extract robust sequence features that are informative for differentiating between performance groups. For future studies, we recommend including background characteristics and timing data in the analysis of process data to further explore their interaction effects on performance.  

Authors

• Qiwei He, Educational Testing Service

• Matthias Von Davier, Educational Testing Service