Knowledge Discovery From Selene Data

• In Event: New Measurement Paradigms: Psychometric Methods for Technology-Based Assessments

Abstract

The educational lunar formation game, Selene, by design and by audience, generates large amounts of data. The Selene system of software captures every player gesture, measures player progress towards a goal every ten seconds and collects a players self-assessment of skill and challenge at regular intervals. Thus, the data collected from a single player session usually numbers in the thousands of records. In its initial development phase, ninety-six (96) individuals played Selene to conclusion. We hypothesize that there is valuable information about learning behaviors buried in this voluminous quantity of data. Subsequently we used unsupervised knowledge discovery techniques look for patterns of player behavior.
The fundamental principle underlying Knowledge Discovery from Data (KDD) lies in the difference between supervised and unsupervised machine learning algorithms. Supervised techniques use a collection of independent variables to predict or classify one or more dependent variables. They constitute the most common and widely used analytic methods. Examples include the various regression techniques, Bayesian classifiers and networks and artificial neural networks. Unsupervised techniques treat all the data in the same way; there are no independent or dependent variables. Unsupervised algorithms generally cluster, or group, data based on similarities and differences among patterns found in the data. Most of the methods involve the computation of some N-dimensional distance metric, such as Euclidean distance, or vector angles. Example algorithms include, K Nearest Neighbor (KNN) analyses[1-3], Learning Vector Quantization (LVQ) [4], Self Organizing Maps (SOM) [5,6] and hierarchical clustering [7].
We analyzed Selene player data with SOM and hierarchical clustering techniques [8]. The data gravitated toward two edges of the SOM map, leaving a band of centroids down the center of the map where no exemplars matched. This suggested the presence of a single strong classifier among the set of exemplar variables. Hierarchical cluster analysis showed that the Selene variable, choice, strongly segmented the data. Analysis of demographics with respect to SOM cluster membership revealed no clear relationship of demographics to Selene game play. In other words, player performance and flow reports seemed independent of player demographics. The subsample of ninety-six (96) players followed the interface prompt by selecting the best choice 70.5 percent of the time. A repeated measures analysis of variance using players’ choice mean score aggregated at four sub-segments indicated choice performance improved significantly over time F(3,81)=10.7, p<0.001, partial η2=0.28. By combining SOM with traditional descriptive and inferential statistics, we identified and substantiated the presence of an interface effect. The choice prompt was designed to lead players toward the game goal of constructing a Moon that replicates correct proportions of Moon material by density. These results indicate that the interface works as designed and should be retained as the game is revised and optimized.
While these results are rudimentary, we feel that they demonstrate the potential value of applying KDD methodology to serious game play data.

Authors

• Ben Hitt, Wheeling Jesuit University

• Debbie Denise Denise Reese, Wheeling Jesuit University - Center for Educational Technologies