Common Learning Analytics for Learning Games

• In Event: Symposium: Games and Simulations: Learning Analytics and Metrics

Abstract

Objective
To develop common learning analytics that can be used across different learning games as measures on the extent to which a learning game requires the to-be-learned knowledge to succeed in the game.

Perspective
As learning games proliferate, we need some way of objectively determining the extent to which the game relies on knowledge. Common measures that (a) relate to learning outcomes and (b) are easily implementable in a game would be useful for numerous purposes. Common measures can serve as learning analytics to compare learning games on can help, for example, teachers, schools, and parents, to at least identify games that require the use of the target knowledge. For developers, such metrics could be used to improve the design of the game with respect to the target knowledge.

Method/Data Sources
To be useful, the measures should be implementable in different kinds of games (i.e., generalizable) and sensitive to the target knowledge. We selected games from our prior research that were considered successful learning games, had a well-designed telemetry (Chung, 2015), and had scores on an external measure of the target knowledge.
We defined two types of measures: Progress-based measures (e.g., max. number of rounds, mean level time) and performance-based measures (no. of correct first attempts, no. of correct solution attempts, and no. of incorrect solution attempts). We then derived those measures for each game using the game’s telemetry. We then correlated the in-game measures to the external measure of the to-be-learned knowledge targeted by each game. The correlation coefficient was interpreted as the degree of knowledge required to be successful in the game.
Seven math games were examined: Three K3 games, N=56 (McCarthy, Li, & Tiu, 2013); three 6th grade games, N=780 (games (Chung, Choi, Baker, & Cai, 2014); and one college game, N = 33 (Chung, 2015).

Results
Significant correlations (p < .05) between the in-game measures and the external knowledge measures are reported below:
Max. no. of levels: K3 (r=.67), 6th (r =.44, .27, .50).
Mean level time: K3 (r = -57, -.28), 6th (-.37, -.29, -49).
No. of correct first-attempts: K3 (r=.42, .39, .63), 6th (r=.55, .31, .50), college (r=.59).
No. of correct solution attempts: K3 (r=.67), 6th (r=.43, .28, .50), college (r=.40)
No. of incorrect solution attempts: K3 (r=-.40, -.50, -.56), 6th (r=-.45, -.10, -.32), college (r=-.39)

The measure that generalized across games was (a) the number of correct first attempts and (b) the number of incorrect solution attempts. We speculate the first attempt acts like an “unexposed item” where the player is encounter a new test item. Support for this idea is seen in one game where feedback was toggled on and off for the same mechanic under different conditions.

Significance
Overall, these results are promising and suggest that the idea of common learning analytics for learning games is feasible. Measures using the same definition were created in games for different ages, mechanics, and topics. Future work will examine these common measures across more games.

Authors

• Gregory K.W.K. Chung, University of California - Los Angeles

• Jeremy Roberts, PBS KIDS Digital