Toward Integrating Data Mining and Knowledge Engineering for Better Student Models

• In Event: 63.059 - Emerging Perspectives on Understanding Learning Behaviors in Digital Environments

Abstract

In this talk, we discuss our work to integrate data mining with knowledge engineering approaches for developing models of students that leverage both the power both of human reasoning and systematic exploration of data sets. In the last few years, there has been continuous debate about whether rational models (sometimes developed using knowledge engineering methods) or data mined models are the best way to develop models of constructs such as student knowledge, domain structure, affect, and gaming the system. Researchers in favor of rational models argue for the greater interpretability and portability of their models; those in favor of data mining argue that their models obtain higher agreement to external measures and that their generalizability is easier to validate.

In this talk, we present work from two intelligent tutoring systems, Inq-ITS and Cognitive Tutor Algebra, that argues that, like peanut butter and jelly, the combination of data mining and knowledge engineering is superior to either alone.

In the first example, we discuss our work to model scientific inquiry skill within Inq-ITS, a performance-based inquiry assessment environment in which students receive feedback as they experiment with simulations for Physical, Life and Earth science. We have built models using data mining alone that predict new students’ inquiry skill effectively. But, if domain experts’ reasoning is integrated into the feature selection process (the part of data mining where variables are chosen for a model), the models have been shown to better predict inquiry skill in new scientific domains.

In the second example, we discuss our work to model gaming the system (succeeding in an online learning system without trying to learn, by misusing the system’s help or feedback) within Cognitive Tutor Algebra, an online problem-solving tutoring system. Data mined models have been shown to do a good job at predicting human coding of gaming, functioning accurately for new students and new lessons within the tutor software. Comparisons of data mined and rational models have revealed that data mined models outperformed rational models at predicting expert labels of gaming behavior. But, a thorough knowledge engineering process, involving interviews with expert coders of gaming behavior, creating models and then discussing those models in detail with those coders and refining the models, has been shown to yield a better model. Finally, a hybrid approach that uses knowledge engineering to determine what atomic aspects of behavior may be relevant, and uses machine learning to search among them, performs better still.

In discussing these two examples, we will illustrate how the combination of human reasoning and data mining is superior for student modeling in online learning than either approach alone. We argue this approach can lead to more robust models of student cognition and affect that work across a variety of learning systems.

Authors

• Ryan Baker, Teachers College, Columbia University

• Luc Paquette

• Michael A. Sao Pedro, Worcester Polytechnic Institute & Apprendis LLC

• Janice D. Gobert, Worcester Polytechnic Institute