Machine Learning Aspects of Computational Psychometrics

Session Type: Coordinated Paper Session

Abstract

Computational psychometrics blends theory-based psychometrics with data-driven approaches from machine learning, data science, and artificial intelligence. Focusing on machine learning aspects, this coordinated paper session highlights five recent developments that harness computational psychometrics to confront educational measurement challenges spanning open item scoring, item calibration, and item feature selection. The first paper uses deep learning to automatically score open-response items on several criteria regarding style and content, and illustrates how such analyses can be implemented in R. Papers two to four aim to drastically minimize sample size requirements for piloting and calibration studies. The second paper uses automated machine learning to construct IRT models based on item-level features to speed up the piloting process and improve model fit. The third paper informs Bayesian estimation of the three parameter logistic model (3PL) with item parameters predicted by neural networks trained on items' linguistic features. The fourth paper presents a likelihood-free, neural-network-based estimation approach for the 3PL that draws on a given item pool to minimize sample size requirements. The fifth paper investigates feature selection procedures in the presence of missing data, comparing the performance of commonly used machine learning techniques (LASSO, Elastic Net) and recently developed approaches based on metaheuristics (Genetic Algorithm).

Sub Unit

• Individual Papers, Innovation Demonstrations, Coordinated Sessions, or Organized Discussions

Session Organizers

• Esther Ulitzsch, University of Oslo

• Dmitry I. Belov, LAW SCHOOL ADMISSION COUNCIL

Individual Presentations

• Application of Deep Learning for the Evaluation of Text Responses - Rudolf Debelak, University of Zurich; Benjamin Wolf, University of Zurich

• AutoIRT: Improving the Fit of IRT Models Using Automated Machine Learning - James Sharpnack, Duolingo; Phoebe Mulcaire, Duolingo; Geoff LaFlair, Duolingo; Kevin Yancey, Duolingo

• Calibration Sample Reduction by Enriching IRT Estimation With NLP-Informed Parameter Predictions - Esther Ulitzsch, Leibniz Institute for Science and Mathematics Education; Dmitry I. Belov, LAW SCHOOL ADMISSION COUNCIL; Alexander Robitzsch, Leibniz Institute for Science and Mathematics Education; Oliver Lüdtke, Leibniz Institute for Science and Mathematics Education

• Neural Networks Approach to Estimate the 3PL in Small Samples - Dmitry I. Belov, LAW SCHOOL ADMISSION COUNCIL; Esther Ulitzsch, Leibniz Institute for Science and Mathematics Education; Oliver Lüdtke, Leibniz Institute for Science and Mathematics Education

• Machine Learning Approaches to Variable Selection With Ignorable Missing Data - Catherine Bain, University of Oklahoma; Dingjing Shi, University of Oklahoma

Discussant

• Hong Jiao, UNIVERSITY OF MARYLAND