Building a Cognitively Sensitive Adaptive Testing System

• In Event: Cognitive Science Goes to College: Scaling Up What We Know About Learning and Instruction

Abstract

Theoretical framework

Technology enables the collection of rich data on student performance but leaves open the question of how to analyze these data to measure learning or growth toward greater expertise. One promising approach uses Bayes nets, which are well-suited to assessing the complex knowledge structures that, according to cognitive science research, are characteristic of subject area expertise. Bayesian analyses can account for the relations among elements of domain knowledge and also enable inferences about student achievement from complex data that might not be possible with existing methods (Mislevey, Almond, Yuan, & Steinberg, 2000; National Research Council, 2001).

Objectives

Our long-term goal is to build a cognitively-sensitive adaptive testing system that maximizes the information about proficiency gained from each item by considering the cognitive structure of the tested domain. To test the "mastery" algorithms for this system, we designed studies with the following objectives: (1) ascertain the extent to which mastery scores predict performance on a target test that student are preparing for, (2) investigate the predictive information added by distractors or wrong answers in multiple choice items, (3) determine the predictive value of mastery scores computed at different points in the course of instruction.

Data

Data were collected from postsecondary students taking online practice tests during a course to prepare for the MCAT (Medical College Admission Test). Item stems, correct responses, and distractors for items in the test database were categorized using a detailed taxonomy of the cognitive domain tested.

We selected a sample of students based on the number of items taken, percent correct, and number of retakes, then recruited 105 students who sent us their MCAT scores.

Methods

To achieve the objectives described above we conducted the following analyses:

(1) compared how well the percent correct for all items taken and the adaptive mastery scores based on those items correlated with actual MCAT scores.

(2) checked the correlation of actual MCAT scores with final mastery scores computed with and without information from distractors.

(3) determined how well percent correct and mastery computed at intervals during the course correlated with actual MCAT scores.

Results

We obtained the following results from the three sets of analyses:

(1) Mastery scores correlated better with MCAT scores (.60) than percent correct did (0.41), and a linear regression showed that mastery scores predicted MCAT scores better than percent correct did.

(2) Including information from distractors did not improve the predictive value of mastery scores.

(3) Mastery scores calculated throughout the course of instruction were higher correlated, better predictors of MCAT scores, and their relative value increased over time, compared with percent correct.

Significance

We expect that the adaptive test system we are building, since it reflects the relational structure of the knowledge domain, will enable better instructional decisions than fixed tests or other adaptive test systems. Our initial findings suggest that using a Bayesian algorithm to consider relational and other information on student behaviors can result in better predictions of performance and learning.

Authors

• Sylvia Tidwell Scheuring, Arroki Inc

• David Michael Niemi, Kaplan

• Larry Rudman