Evaluating Consistency of Behavioral Patterns Across Multiple Tasks Using Process Data: An Empirical Study in the Program for the International Assessment of Adult Competencies

• In Event: Response Process Data Analysis

Abstract

The digital assessment platform provides the possibility to collect process data such as action sequences and response times along with the responses to each task. Evaluating the behavioral consistency across items, rather than single items, renders possible capturing and modeling person-related latent characteristics, in addition to the measurement by response accuracy. This paper introduces two approaches that could help assess the consistency of test-taking behaviors across items by using both aggregate-level response process variables and finer-grained information extracted from action sequences. We illustrate these two approaches and their utilities by conducting an empirical study on a set of seven complex problem-solving items in PIAAC. Recommendations for assessing behavioral consistency in different research settings and directions for future studies are discussed.

Authors

• Qiwei He, Georgetown University

• Dandan Liao, McKinsey & Company

• Hok Kan Ling

• Hong Jiao, University of Maryland