How Small Is Too Small? Sample, Item, and Correlation Size in Rasch Out-of-Sample Classification Error

• In Event: Rasch Models: Considerations and Use

Abstract

This simulation study investigates the factors that affect various methods’ ability to estimate out-of-sample classification error of binary Rasch-based assessments. The goal is to identify how to accurately characterize the misclassification rate of an assessment with a pilot study-sized sample. Five resampling methods compare the classification error of the pilot sample against a secondary sample across three factors: initial sample size, number of questions, and correlation between latent trait and classification variable. Binomial tests and multiple graphs are used to determine which methods under which factor combinations best approximate the out-of-sample error. Results indicate that as sample size and number of items increase, out-of-sample error estimates become more accurate, but correlation to the classification variable is the most important factor.

Author

• Anthony Raborn, University of Florida