Building Machine Learning Models to Detect Item Response Theory Item Misfit (Poster 15)

• In Event: AERA Poster Session Two
  In Poster Session: Measurement Theory and Application 1

Abstract

One important facet of performing valid IRT-based inferences is agreement of the data with the parametric form specified by the model. Using 3900 operational IRT calibrations from a large-scale educational survey, machine learning (ML) techniques created models that successfully detect items which exhibit misfit to the IRT model. Features used were based on the pseudocounts, including G^2/X^2, mean deviation and root mean square residual, and measures based on comparing IRF to nonparametric curve based on kernel smoothing of the pseudocounts. ML models, including regularized logistic regression, decision trees, random forest and support vector machines, classified the item as Fit or Misfit. Issues such as out-of-sample validation and imbalanced data are discussed. Results indicate high accuracy (>95%) in classifying items.

Author

• John R. Donoghue, Educational Testing Service
  Presenting Author