Extracting Game-Based Indicators Using Distance to Optimal Performance

• In Event: Game-Based Indicators of Learning Processes: Extraction Methods, Validity Evidence, and Applications

Abstract

Objectives
The goal of this study was to extract meaningful and interpretable game-based indicators (GBIs) from students’ gameplay data. These GBIs will then be used as inputs to computational or statistical models to infer students’ knowledge and skills.

Background
In machine learning, computational models are usually trained by minimizing a cost or distance function, which provides information about how close an outcome is relative to an optimal outcome. Analogously, when interpreting gameplay data, we can define a function that represents the “distance to optimal performance,” with optimal performance defined as the optimal student action, given a current game situation, and given the goal of the game. Optimal actions will decrease this distance and suboptimal or wrong actions will increase it. Therefore, patterns of distance changes, together with game context, provide insights into students’ thinking that will inform the creation of GBIs for further processing.

Methods
We developed a distance metric that encodes the accuracy of code edits in a block-coding game. This distance metric is based on the Damerau-Levenshtein edit distance, defined as the minimum number of simple edit operations (insertion, deletions, substitutions, transpositions) required to transform a given code into another (optimum) code. With this distance function and its change over time, we derived GBIs that encoded behavioral patterns related to students’ thinking in general, and, specifically, related to computational thinking (CT) constructs, such as abstraction, decomposition, modeling, algorithmic thinking, generalization and evaluation.

Data sources
We collected gameplay data from 25 students (10 first-, 15 third-grade) playing the block-coding game codeSpark Academy over a period of six weeks, with weekly 45-minute online teacher check-ins, where questions were answered and new levels were explained. For external validation, we administered the TechCheck assessment as a pre- and posttest (Relkin et al., 2020) of CT skills.

Results
Students in the codeSpark curriculum improved on TechCheck in all above CT constructs by 9.87% on average. In contrast, a non-coding control group engaged in usual classroom activities over a comparable period of time improved on average by 1.80%.
Further analysis showed that GBIs can be derived from game telemetry to describe in-game behaviors related to cognitive processes and that, once defined, they can be automatically detected and extracted from gameplay data. GBIs of numbers of continued editing and reload events correlated with pre-post TechCheck change scores at r=-.53 and r=-.51, respectively (p < .05). Additional in-depth analyses and interpretations will be provided.

Significance
The introduction of distance functions that are rooted in optimal performance provides a reliable and well-defined way of assessing gameplay performance. Multiple distance functions that evaluate separate, specific performance dimensions (e.g., time, accuracy, content, structure, fluency, or frequency) can be defined. For tasks where optimal outcomes or actions are not clearly defined, such as tasks requiring creativity, other function criteria can be introduced, such as the number of different ideas introduced. Finally, the resulting distance values can be used to define GBIs which provide the relationships to higher level constructs of knowledge and skills.

Authors

• Markus Iseli, University of California - Los Angeles

• Tianying Feng, National Center for Research on Evaluation, Standards, and Student Testing

• Emily Relkin, Tufts University