Executive Assessment Based on Multistage Testing With Cross-Sectional Routing

• In Event: 72.016 - Innovations in Computerized Test Designs and Implementations

Abstract

In 2016, the Graduate Management Admission Council® (GMAC®) launched the Executive Assessment (EA) test, which it developed to evaluate a candidate’s readiness for executive MBA degree programs. The EA test consists of three sections, each of which is designed to measure (1) quantitative reasoning skill, (2) verbal reasoning skill, and (3) integrated reasoning skill, respectively. A majority of the executive MBA candidate population consists of working professionals, and in order to meet the needs of both the candidates and the programs, it was critically important for the EA test to strike the right balance between the test-taking experience and measurement efficiency/precision. GMAC’s response was to employ new multistage testing (MST) with cross-sectional routing in the design of the EA test.

MST—often viewed as a specialized version of computerized adaptive testing (CAT)—offers distinct advantages for both the test developer and the test taker over typical CAT formats. For example, MST offers test developers more control over test construction in terms of possible test forms and content combinations. At the same time, it enables test takers to move back and forth among test questions within each test section. A substantial tradeoff, however, between MST and traditional CAT, is the reduced adaptability of the test, especially when a test has a minimal number of stages. In a typical MST administration, for example, the first stage usually is a routing stage in which all test takers see test items with the same average difficulty level. Assuming a test has three test sections—with each one measuring different traits—and each section consists of two stages with the first stage being a routing stage, then only three of the six stages (i.e., about a half of the test) would be adaptively administered. The other half would be administered as a linear test. If multiple test sections measure different but moderately or highly correlated traits, then a score estimate for one section might be adequate for adaptively selecting item modules for following sections without needing to administer routing stages repeatedly for each section. This strategy is called cross-sectional routing (CSR). It was investigated comprehensively under various conditions with three key factors: (1) MST design (e.g., number of modules in the first stage), (2) correlation among measured traits, and (3) standard error of measurement of score estimates that were used for routing. In MST with CSR, the correlational relationship among sectional scores is used only for routing and not for score estimation for each section. In doing so, each section retains its unidimensional properties as a measure for each trait. The result of study suggests that employing MST with CSR can meaningfully improve the overall measurement efficiency, especially when the test is short. The study also introduces the real-world implementation of the new MST with CSR with the EA test.

Author

• Kyung (Chris) T. Han, The Graduate Management Admission Council