Poster #90 - Measuring distance between solutions in an engineering game for children

• In Event: Poster Session 08

Abstract

Online games allow us to learn about players’ evolving conceptions through anonymous gameplay data collected in the background (Mislevy et al., 2014; Roberts, Chung & Parker, 2016). Rating the quality of a player-submitted solution is a necessary step in any game, but in a learning game, tracking successive submissions may also allow us to follow players’ conceptual change and better understand how well the game is supporting learning (Hao, Shu & von Davier, 2015).

We use gameplay data from an engineering game, Base Builder, designed for ages 4 to 7, to propose a rating system for player-submitted solutions. In Base Builder, children are asked to drag pieces from a menu onto a canvas in order to build a structure that will protect an astronaut figurine. The game offers progressively more difficult challenges and a scaffolding system with visual tips and messages of encouragement from a character. After a user submits a solution, the game simulates how the proposed structure would fare under different weather events. If the proposed goal (e.g. protect the astronaut from rain) is achieved, players are allowed to continue to the next level. We use the game’s proposed solution, which is visible through a scaffolding system, as a comparison point for user submissions (Figure 1).

First, an algorithm is proposed to compute the distance between any two solution attempts. The algorithm attempts to match any piece of the structure of the first solution to the closest piece in the second solution. After matching is complete, the distance between coupled pieces is extracted and a regularization function is applied. The purpose of the regularization function is to assign different weights to varying offsets between pieces. For instance, there might be a larger penalty applied to the first 100 pixel offsets than to larger offsets. The total distance is computed by the sum of the regularized distances between pieces. Several variations to the algorithm are also considered, for instance adding penalties for missing or extra pieces.

Once an algorithm has been established to compare generic solutions, we are able to compare each successive attempt to a reference solution, for example, the game’s proposed solution, or the user’s last submission, presumably their most evolved. Comparing user submissions to their own earlier attempts allows us to track the evolution of their thinking, while comparing the user’s last submission to an ideal solution allows us to measure their progress toward mastery.

To continue working towards evaluating players’ progress towards mastery of the game’s engineering concepts, we need to consider that there may be more than one ideal solution. Any possible ideal solution in Base Builder might involve not only structural soundness (e.g. stability and symmetry) but also factors such as the right physical materials to avoid deterioration (e.g. damage from weather events) if the structure existed in the real world. We plan to engage engineering experts in establishing a set of rules to identify all possible solutions in order to compare the player’s attempts to all of them.

Authors

• Cosimo Felline, PBS KIDS
  Presenting Author

• Silvia Lovato, Public Broadcasting Service

• Jeremy Roberts, Public Broadcasting Service