Case Study of a Transition From Computerized Adaptive Testing to Pallet Assembly

• In Event: 72.016 - Innovations in Computerized Test Designs and Implementations

Abstract

Pallet assembly (PA) is a test administration model in which a collection of many equivalent test forms (a “pallet”) is constructed using automated test assembly methods based on form- and pallet-wide objectives and constraints. At testing time, a test form is randomly selected from the pallet for administration to each examinee. The present platform uses mixed-integer programming and the CPLEX optimizer to assemble forms.

The North American Pharmacy Licensure Examination (NAPLEX) offered by the National Association of Boards of Pharmacy (NABP) is a credentialing exam designed to measure whether a candidate has mastered sufficient competence in pharmacy to be a licensed practitioner. The examination covers a content outline across seven primary domains with additional constraints for other types of content. In addition, approximately two-thirds of the NAPLEX consists of item clusters surrounding a clinical profile. The NAPLEX currently is administered as a fixed-length computerized adaptive test (CAT) containing 150 operational items. In the fall of 2016, the NAPLEX will transition to the PA model, in which one form with 200 operational items will be randomly administered to each candidate from a pallet of pre-assembled forms.

Though CAT offered NABP some distinct advantages over fixed-form models, such as pool-based exam delivery, good score precision across the scale, and ease of item masking when items became technically obsolete, these advantages mitigated over time as programmatic goals and test security concerns evolved. First, due to the large proportion of clinical profiles that contain items of variable difficulty, the exam was not fully adaptive, and hence the full benefits of a CAT could not be achieved, bringing into question the cost-effectiveness of that testing model. Additional drawbacks of CAT for the NAPLEX program included: proneness to item overexposure, item pool under-utilization, and larger-than-desired form overlap. All of these drawbacks were relieved by PA without sacrificing pass-fail classification reliability.

The objective of this paper is to provide a case study outlining the methods that were used in the planning and evaluation process leading to the deployment of a pallet-based test delivery model for the NAPLEX. Using results from CAT administrations over the last few years, current characteristics of the NAPLEX were compared with expected characteristics from several different test assembly objective and constraint combinations for both a 150-item and 200-item test. Test characteristics examined for the CAT and PA models included expected measurement precision, pass-fail classification consistency, item usage and exposure, and form overlap. The assembly objectives and constraints included rigid content constraint compliance, test information (TIF) and test characteristic curve (TCC) specifications, and degree of item usage. Initially, an objective that minimized tolerance around both a target TIF and TCC provided the best results. It offered better measurement precision than the 150-item CAT while also maintaining desired levels of item exposure and form overlap. During the planning and evaluation process, however, there was a change to the NAPLEX exam blueprint. It became necessary to once again change the optimization objective in order to continue to obtain comparable forms of the desired precision while still controlling for item exposure.

Authors

• Jennifer Davis, National Association of Boards of Pharmacy

• Paul Edward Jones, National Association of Boards of Pharmacy

• Maria Incrocci, National Association of Boards of Pharmacy