Exploring the Sequential Patterns in Program for the International Assessment of Adult Competencies Problem-Solving Environments by U.S. Adults' Employment-Related Variables

• In Event: Understanding the Digital Skills of Adults and How to Improve Them

Abstract

Objectives and Theoretical Framework
Adult assessments have evolved to keep pace with the changing nature of adult literacy and learning demands. Specifically, problem solving in technology-rich environments (PSTRE) assessment in PIAAC renders it possible to measure how well adults process, analyze, and address problems for specific goals in a computer-based environment. However, U.S. test takers on average scored lower than test takers from other countries in the PSTRE domain (OECD, 2013, p. 11; Rampey et al., 2016). Some facts about specific subgroups of U.S. test takers are also concerning, regarding age, education, employability, etc. (Goodman, Sands, & Coley, 2015; OECD, 2013, p. 21; Vanek, 2017).

Data and Methods
In this study, we focused on the data from background questionnaire and log files for the U.S. population in the main study of 2012 PIAAC. This study explored the association between process data and employment-related variables from three aspects. First, we used regression analysis and n-grams to extract features from process data that discriminate the best between subgroups with different employment-related variables. Second, we performed cluster analysis using features extracted from process data, then compared the background profiles among clusters. Lastly, we examined the consistency of test takers’ response behaviors across multiple items, and explored the differences in background profiles for different consistency groups.

Results
Similar to what He and von Davier (2015, 2016) found, test takers who had higher income, work-related skill use, and education level demonstrated clear subgoals in their process of solving the item. Some key actions that demonstrated evidence of necessary steps to solving the task were found more often in the groups with higher income and work-related skill use. Groups with lower income and work-related skill use, however, took more actions which were either redundant or indications of failing to complete the request of the task.

It was found that more actions and longer response time in general were associated with higher PSTRE scores, and such pattern was more evident when the test takers did not answer correctly to the example item in the study. In other words, it was possible to obtain a correct answer with different strategies, but when test takers answer incorrectly, process data could be informative about the extent to which interventions would be needed. Test takers who did not put much effort in solving the item tended to have semi-skilled or elementary occupations, lower monthly income, lower work-related skill use, higher age, and lower education. They also tended to make less effort consistently on other items. This group of test takers was considered in need of further education or intervention.

Significance
To summarize, this study provides critical evidence of relationships between employment-related background variables and sequential patterns in PSTRE based on the U.S. sample in PIAAC. It also provides information to education policy makers to explre the test-taking behaviors by different employment-related subgroups, thus helping to find an optimal solution to improve their PSTRE skills via a tailored approach.

Authors

• Dandan Liao, Cambium Assessment, Inc.

• Qiwei He, Educational Testing Service

• Hong Jiao, University of Maryland-College Park