What It Really Takes to Implement a Linear-on-the-Fly Test (LOFT) Exam

• In Event: 72.016 - Innovations in Computerized Test Designs and Implementations

Abstract

Implementing a linear-on-the-fly test (LOFT) design for an operational testing program may seem like a trivial and straightforward way to satisfy two goals simultaneously—namely, to administer test forms with comparable statistical and content characteristics to all test takers, and to control item exposure and thereby enhance test security by administering a unique test form to each person. But a LOFT test design involves much more than the simple random selection of items from an item pool. Successful implementation of a LOFT design requires a substantial amount of work prior to the test’s deployment.
For example, the test form administered to each person must conform exactly to the test blueprint (or content specifications) in place for the test. Historically, however, test forms for some programs may have been constructed by subject matter experts who used their best judgment to also balance additional characteristics, such as age of item, or item content, at some detailed subdomain level that was not reflected in the blueprint. Thus, it may be necessary to mirror those subtle human judgments (i.e., additional constraints) in the new LOFT forms.
Specification of the statistical constraints can take many forms, but the primary goal is to ensure that the test-taking experience is the same for each person. If the item statistics are based in Item Response Theory, then the statistical constraints are often expressed in terms of a test information function. In all cases, the details of these constraints must be determined— Comparable information near the cut-score? Comparable information across the full score range? How much variation in information to permit across candidates?). The constraint details, then, ideally should correspond with the statistical characteristics (if available) of historical test forms.
It is unlikely that a large, high-stakes testing program would make an entire item bank available for a LOFT test. More likely, a subset of the item bank would be live at any one time, with the remainder of the bank held in reserve to be used during the next test administration window (with little or no overlap across successive LOFT item pools). In this scenario, the LOFT item pool must be “sculpted” in a manner that ensures that adequate items exist in each content area and with appropriate item statistics so that the goals of the LOFT test can be met in successive test administration windows.
Importantly, the successful implementation of a LOFT design requires that exam administrations be simulated in advance to ensure that the specified combination of content constraints, statistical constraints, and item pool can yield valid test forms when used operationally.
This paper presents a description of how a LOFT design was successfully implemented with a large, high-stakes testing program and will provide guidance for how that experience can be generalized and applied to other testing programs. Details of the procedures followed at each step in the process will be provided, and the results from example simulations will be presented and evaluated.

Author

• Kathleen A. Gialluca, Pearson