Poster #236 - Deriving Learning-Related Measures From Game Telemetry: Detecting Children’s Alternative Conceptions of the Pan Balance

• In Event: 1-061 - Poster Session 02
  In Poster Session: PS 02 Section - Technology, Media & Child Development

Integrative Statement

A critical issue in the description of learning processes in games is the measures: What is their definition, what constructs do they purport to measure, and what is the empirical evidence supporting its interpretation? In this study we report on a theory-driven approach to the development of measures (vs. a data mining approach to discover “interesting” patterns). This approach leads to measures that are interpretable, more likely fit into a larger theoretical framework (vs. patterns discovered ad hoc) and are likely to be sensitive to learning.

METHOD

We began the measures design process with theory and prior empirical work (Metz, 1993). Metz identified the strategies that children (ages 3 to 5) use while solving pan balance problems. We focused on the strategy “Displace Elements Across Pans” (DEAP) (defined as attempting to balance the pan balance by moving weights from the up-pan to the down-pan) because of its prevalence.
The first step in the measures development was to map the interaction space of a game to document what game elements players can interact with. Once mapped, we identified the key decision points in the interaction (Reisig, 2013). We then incorporated theory and prior research by referring to the literature for guidance on what children know about the topic and how they are likely to express their understanding, and reconciled the theory with the game interaction space to operationalize a process that unfolds in the game—the algorithm.

For the games in this study, the gameplay allowed us to compute the number of times a child placed a weight on the low pan or subtracted weight from the high pan. These in-game actions and other gameplay information, such as number of times they received feedback about this error and total play time, were incorporated into the algorithm to detect when children were using the “Displace Elements Across Pans” strategy.

RESULTS

Correlations were computed between (a) the DEAP strategy measure and external measures of understanding weight, and (b) between the DEAP strategy measure computed for each game. In game A, the strategy measure was negatively related to the posttest knowledge of weight (r(52) = -.36, p < .01) and children’s gain in weight knowledge (r(52) = -.34, p < .05). In game B, there were no significant correlations between the strategy measure and any outcome measure. However, there was a significant correlation between the strategy measures for game A and B (r(52) = .33, p = .02), suggesting that children carried the DEAP strategy across games.

SIGNIFICANCE

The importance of this work is that it describes a principled way of deriving measures from gameplay. Mapping the set of permissible interactions and gameplay flow can be used to identify key decision points in the game and key gameplay paths. This information facilitates both the generation of hypotheses of a learner’s state as well the behavior that would constitute evidence of that state, which together provide the basis for developing the algorithm.

Authors

• Jeremy Roberts, PBS
  Presenting Author

• Gregory Chung, UCLA-CRESST
  Non-Presenting Author

• Tianying Feng, UCLA/CRESST
  Non-Presenting Author

• Claudia Riveroll, UCLA/CRESST
  Non-Presenting Author

• Elizabeth Redman, UCLA-CRESST
  Non-Presenting Author

• Katerina Schenke, UCLA-CRESST
  Non-Presenting Author

• Anne Lund, PBS KIDS
  Non-Presenting Author

• Jennifer Rodriguez, PBS KIDS
  Non-Presenting Author