Search
Program Calendar
Browse By Day
Browse By Time
Browse By Person
Browse By Room
Browse By Committee or SIG
Browse By Session Type
Browse By Keywords
Browse By Geographic Descriptor
Partner Organizations
Search Tips
Personal Schedule
Sign In
Computer-based testing allows researchers to capture both response data (if the respondent solves a problem) and response process data (what steps a respondent uses to solve a problem and how much time s/he requires to do so). A variety of timing and process data, such as sequences of actions, is recorded in computer generated log files that complement information on test performance data. Such information is particularly valuable when examining interactive problem-solving tasks, which have been increasingly used in large-scale assessments such as the Programme for International Student Assessment (PISA) and the Programme for International Assessment of Adult Competencies (PIAAC). Because problem-solving items are often designed within a specific context, the interpretation of process data for any individual item is highly dependent on to the particular task that is being analyzed. Although exploring sequence patterns for a specific item is a good starting point, investigation of generalized patterns across multiple tasks bears the promise of identifying factors that are associated with test takers’ problem-solving behaviors across a variety of contexts and frames.
This paper illustrates an approach to generalize action patterns across multiple tasks using data from the problem-solving in technology-rich environment (PSTRE) items that were administered in PIAAC. Motivated by sequence mining methods in natural language processing and biostatistics, we applied the longest common subsequence (LCS) method to derive the degree of similarity between an individual’s action sequences and predefined optimal problem-solving sequences.
As results from the present study, consistent behavioral patterns were found for a large number of respondents across countries. More consistent patterns were observed in groups performing either extremely well or extremely poorly on the test. The similarity as well as the consistency of similarity of respondents’ problem solving sequences against the predefined sequences make significant influence on the test takers’ problem solving proficiency level. The higher degree of similarity gains, the higher problem solving proficiency level is expected. Within the moderate and high similarity groups, the less consistent groups apparently show lower proficiency level. Interactions between background variables and similarity of sequences were also found in this study. For instance, the younger respondents were found more likely to follow the predefined action sequences when solving digital tasks. Male respondents were more likely to choose similar strategies as predefined ones, while female respondents showed more consistent patterns across similarity groups.
These results suggest a promising avenue for further exploring the relationship between action sequences across multiple tasks and proficiency estimates in computer-based large-scale assessments. Further, process data also help uncover how people make success or failure when solving problems, which provides important clues for instructions and learning design for low-performing adults. Mapping sequence patterns with background variables, such as gender, language and culture related factors enables the identification of subgroups who need special supports to improve their ability across countries. With the help of advanced methods in analyzing new data source under the computer-based tests, sustainable development in education, especially in lifelong learning will be easily tracked and tailored in an adaptive learning environment.