From Artificial Intelligence to Intelligent Assessment

• In Event: Cognitive Models for Assessment

Abstract

This talk will discuss how generative cognitive process models such as those used in artificial intelligence and cognitive science can be used as psychometric models to measure different interacting components of cognition. The approach will be demonstrated with a Markov Decision Process model which has been adapted to measurement for complex tasks.
Computational cognitive models have been researched for decades in the fields of artificial intelligence (AI) and cognitive science. While the traditional AI approach is unconcerned with how well the models reflect actual human cognition, in the field of cognitive science the objective is to build models that inform our understanding of how the human mind actually works. These models, therefore, are likely to contain components and parameters that are interpretable as construct of interest to education and psychological measurement.
Latent trait models used in psychometrics depend upon the assumption that we can specify a valid mathematical relationship between the latent constructs we wish to measure, such as inquiry ability, and the data generated by human performances on specific tasks. As cognitive process models share this same goal, it seems natural that such cognitive models might be candidates for use in psychometrics. To work as a latent-trait measurement model, the cognitive model must be able to generate probabilities for any action that a student might take as a function parameters which represent the constructs we wish to measure. Many of the models used in cognitive science or AI produce only a predicted action, rather than probabilities for all available actions. Thus modifications must be made to expose the probabilities that often exist within the model functions. A further problem is that cognitive process models often utilize parameters at a very low level, such as memory-retrieval time latencies, which are unlikely to be of interest in either educational or psychological assessments. Thus we seek cognitive models that explicitly represent constructs at the granularity of our intended measurement.
The Markov decision process (MDP) has been used in both artificial intelligence and in cognitive science as a model for either ideal or human decision making (Puterman, 1994; Baker, Saxe & Tenenbaum, 2011). As a demonstration of use of cognitive process models for measurement, we will show how the MDP can be used for the inference of beliefs, motivation or strategic problem-solving ability (Rafferty, LaMar & Griffiths, 2015). Two applications of the model will be demonstrated. First, modeling an educational game will show how the MDP can be used to estimate both student misconceptions and ability. The second application will demonstrate use of the MDP to detect inquiry strategy in a simulation-based science assessment.

Author

• Michelle LaMar, Educational Testing Service