Revision and Review Behavior in Large-Scale Computer-Based Assessments: An Analysis of NAEP Mathematics Process Data (Poster 13)

• In Event: Excellence in Education Research: Early Career Scholars and Their Work

Abstract

The rising role of computer-based testing (CBT) preordains the importance of cost-benefit analyses on many design features, in terms of their psychometric and practical impacts, to inform test developers as they make this assortment of decisions on how a CBT should be delivered, depending on the specific context and needs. One of the important design features for a CBT is whether response review and revision should be allowed. Answering this question requires a thorough understanding of different types of revision/review and test navigation behavior, as well as their relationships with performance, motivation, and demographic background. We explore examinee test revision and review behavior in the computer-based version of the 2017 NAEP 8th Grade Mathematics Assessment, based on the NCES restricted-use process, response, and survey data. Combining latent variable models and machine learning techniques to analyze unstructured revision and review data, we propose new tools for clustering and extracting features from test revision and review log data. We present preliminary results from the empirical analysis of the NAEP data employing these tools, which identifies key dimensions and common prototypes of visit behavior and test navigation patterns. The findings are expected to offer insights into how different types of examinees interact with the test allowing review and revisions, potentially informing future test design and administration strategies, as well as instructions that promote effective problem-solving.

Authors

• Susu Zhang, University of Illinois at Urbana-Champaign

• Shiyu Wang, University of Georgia