Editorial Changes and Item Performance: Implications for Recalibration and Pretesting

• In Event: Impacts and Innovations in Assessment Methods for Professions Education

Abstract

Previous research on the impact of text and formatting changes on test-item performance has produced mixed results. The present study investigated the effects of 7 classes of stylistic change on item difficulty and response time. Each of 65 item pairs (original, revised) were assigned to approximately 480 examinees per item in a randomized experiment. None of the changes affected item difficulty, while one class of edits – changing an item from an open lead in (incomplete statement) to a closed lead in (direct question) – did result in slightly longer response times. These results have implications for the conventional practice of re-pretesting (or recalibrating) items that have been subjected to editorial changes.

Authors

• Heather A Stoffel, National Board of Medical Examiners

• Mark R. Raymond, National Board of Medical Examiners

• S. Deniz Bucak, National Board of Medical Examiners

• Steven Arthur Haist, National Board of Medical Examiners