Cross National Comparisons in Problem Solving Sequence Patterns Using Process Data

• In Event: Advanced methods in cross national and cultural comparisons using process data

Proposal

Technical advances in computer-based testing provide possibility in recording a variety of timing and process data such as sequences of actions in log files accompanying test performance data when test takers respond to items. With the help of process data, sequence patterns could be identified in test takers’ problem solving behaviors by different performance groups. Background variables are of importance to further exam consistency of test takers’ behavior patterns across items and countries in large scale assessment. The primary objectives of this study are to (1) investigate the effects in adding background variables as additional features in identifying malleable factors associated with test takers problem-solving proficiency, and (2) explore the relationship between sequence patterns and background questionnaires.

This empirical study used log file data of 8,599 test takers from six countries as well as their background and cognitive dataset collected in Programme for International Assessment of Adult Competencies (PIAAC). We employed different regression analyses (i.e., linear regression, logistic regression and Poisson regression) to investigate the relationship between different variables of interest according to the type of the response variables. The results show that the test takers with high information technology communication (ITC) skills at home are more likely to get higher problem solving performance across countries. Gender has significant differences in digital task solving strategy. High performing countries are more likely to use tools and have clear strategy in solving the tasks compared with low performing countries. Older people, female and low ITC at home or work need to get more intervention to improve their problem solving skills. In conclusion, background variables play an important role in identifying subgroups that need special supports when mapping with process data. Consistent problem solving patterns were found across items and countries in subgroups, which help generalize our findings in a broader scope.

Authors

• Qiwei Britt He, Educational Testing Service

• Hok Kan Ling, Columbia University

• Jingchen Liu, Columbia University

• Zhiliang Ying, Columbia University