Detecting Game Player Goals With Log Data

• In Event: Technical Issues in Performance Scoring

Abstract

Purpose
This paper draws on Evidence-Centered Design (ECD; Almond, Steinberg, & Mislevy, 2002), exploratory data analysis (EDA; Jones, 1987), and educational data mining (EDM; Romero, Ventura, Pechenizkiy, & Baker, 2010) to develop and compare two methods for assessing players’ goals within a game.

Theoretical Perspective
Games and simulations are appealing as performance assessment tools in part because they provide fine-grained records of actions. However, the processes for the creation and application of scoring rules with log files to produce observable values to inform measurement models continues to challenge the game-based assessment community.

Much of the evidence used to assess skill proficiency and player attributes assumes that individuals are working towards a specific goal within a game. However, research suggests at least four different types of player goals: achievement, exploration, socializing, and imposition on others (disrupting play of others) (Bartle, 1996). Misunderstanding players’ goals can lead to misinterpretation of evidence within a game. Measurement of engagement, persistence, and creativity, for example, often presuppose a particular goal.

Methods
First, individual detectors for three game goals (quest completion, exploration, and socialization) were developed using Classification and Regression Tree techniques. Logs were hand coded for the presence or absence of specific goals. Potential features in the logs that might indicate goals were identified and a CART analysis using the J48 algorithm was employed to create a decision tree and rules for each goal. Cross-validation using a split-sample technique was used to determine whether these rules generalized to another sample.

Separately, a two-step cluster analysis was performed in order to compare results and groups. First, individual actions were clustered in an attempt to reduce the total number of actions using k-means clustering. Second, individuals were clustered using a fuzzy clustering method that allowed for individuals to belong to several clusters.

Log files of 500 Poptropica® players, aged 6 to 14 years and 49.1% male/ 50.9% female were examined for this project. Poptropica® is a virtual world in which players explore “islands” with various themes and overarching quests that players can choose to pursue. The quests generally involve completion of 25 or more steps (for example, collecting and using assets) that are usually completed in a particular order. Apart from the quests, players can talk to other players in highly scripted chats, play arcade-style games head-to-head, and spend time creating and modifying their avatar.

Results
The detector for quest completion led to the best results among the detectors (kappa=.63, A’=.93), while the detector for socialization had the worst (kappa = .47, A’ = .78). The cluster analysis first revealed that the fuzzy clusters worked better with individual events as indicators than the clustered event types. The clusters of individuals revealed that a four cluster solution was optimal, but that individuals frequently overlapped between clusters.

Significance
This work demonstrates two methods for identifying in-game indicators for a construct and highlights how these can be applied to determination of goal, an important pre-requisite for further interpretation of log file data.
 

Author

• Kristen E. Dicerbo, Pearson