Understanding Simulation-Type Items Using Eye Movement and Log-File Data

• In Event: Response Process Data Analysis

Abstract

Technology-Based Assessments (TBAs) can now facilitate items involving complex simulation-type tasks. While analysing the accuracy of student responses (product data) to such items is valuable, process data offers additional insights into how the responses were produced. Process data in the form of eye movements and other log-file variables were collected from 24 participants who completed five PISA 2015 simulation-type items. Results indicated that test-takers who received full credit for their performance on these items paid more attention to relevant areas of the simulation than those who did not receive full credit. However, this finding was not consistent across all tasks, suggesting that operationalising test-takers’ behaviour and performance in simulation-type items may not be as straightforward as expected.

Authors

• Paula Lehane, Dublin City University

• Michael O'Leary, Dublin City University

• Darina Scully, Dublin City University