Using Process Data to Identify Consistent Patterns Across Problem-Solving Tasks in the Program for the International Assessment of Adult Competencies

• In Event: Understanding the Digital Skills of Adults and How to Improve Them

Abstract

Objectives and Theoretical Background
Computer-based testing allows researchers to capture both response data (if the respondent solves a problem) and response process data (what steps a respondent uses to solve a problem and how much time s/he requires to do so). A variety of timing and process data, such as sequences of actions, is recorded in computer generated log files that complement information on test performance data. Such information is particularly valuable when examining interactive problem-solving tasks, which have been increasingly used in large-scale assessments such as the Programme for International Student Assessment (PISA) and the Programme for International Assessment of Adult Competencies (PIAAC). Because problem-solving items are often designed within a specific context, the interpretation of process data for any individual item is highly dependent on to the particular task that is being analyzed. Although exploring sequence patterns for a specific item is a good starting point, being able to generalize patterns of responses across multiple tasks bears the promise of identifying factors that are associated with test takers’ problem-solving behaviors across a variety of contexts and frames.

Data and Methods
This paper illustrates an approach to generalize action patterns across multiple tasks using data from the problem-solving in technology-rich environment (PSTRE) items that were administered in PIAAC. Motivated by sequence mining methods in natural language processing and biostatistics, we applied the longest common subsequence (LCS) method to derive the degree of similarity between an individual’s action sequences and predefined optimal problem-solving sequences. The mean value and standard deviation of the degree of similarity across items were defined as similarity and consistency of similarity, respectively.

Results
The study found consistent behavioral patterns for a large number of respondents across countries. More consistent patterns were observed among individuals performing either extremely well or extremely poorly on the test. The similarity as well as the consistency of similarity of respondents’ problem-solving sequences is significantly correlated with test takers’ problem-solving proficiency level. Within moderate and high similarity groups, individuals who are less consistent in their problem-solving strategies display lower levels of proficiency. Interactions between background variables and the extent to which respondents fall in the high/middle and low similarity groups was also found. For instance, younger respondents were more likely to follow predefined/optimal action sequences when solving digital tasks. Male respondents were more likely to choose predefined optimal strategies, while female respondents appeared to be more consistent in their responses.

Significance
These results suggest a promising avenue for further exploring the relationship between action sequences across multiple tasks and proficiency estimates in computer-based large-scale assessments. Meanwhile, the LCS approach proposed in this paper also provides evidence of advantages in process data in item quality control thus helps item improvement.

Authors

• Qiwei He, Educational Testing Service

• Francesca Borgonovi, UCL Institute of Education

• Marco Paccagnella