Game-Based Measures of Implicit Learning

• In Event: 43.015 - Game-Based Assessment: How Has the Field Matured Over the Past 10 Years?

Abstract

This poster describes the evolution of methods of measuring implicit learning in gameplay. We will present results from our analysis of implicit understandings of physics understandings in Impulse and Quantum Spectre gameplay (Rowe, Asbell-Clarke, Baker, et al., 2017) to implicit computational thinking in three Zoombinis puzzles (Rowe, Asbell-Clarke, Gasca, & Cunningham, 2017). We are currently incorporating multimodal datastreams into the measurement of implicit learning in Impulse (Dahlstrom-Hakki, Asbell-Clarke, & Rowe, in press).

Players can build implicit knowledge of challenging concepts when playing digital learning games. Learners may demonstrate this knowledge through behaviors that they are not yet able to express formally (Polanyi, 1966; Sternberg, 1996). This is referred to as implicit knowledge. Game-based learning assessments (GBLA) show promise as a method of measuring implicit learning (i.e., changes in implicit knowledge over time) by avoiding jargon, construct-irrelevant material, and test anxiety which can make traditional assessments challenging (Shute, Masduki, et al., 2010).

Our emergent approach to creating measures of implicit learning, which we have refined across 5 games, combines hand labeling of gameplay with educational data mining (EDM) techniques. Specific EDM methods were chosen based on the nature of the game. For Impulse and Zoombinis, games with dynamically changing game states and a large number of solution paths, we build automated tools that use gameplay data to provide information about players’ implicit learning by:
1. Hand-labeling gameplay behaviors consistent with the construct(s) we want to measure.
2. Merging labels with gamelog data.
3. Distilling gamelog data into features useful for measuring specific gameplay behaviors.
4. Building detectors of players’ implicit learning from the gameplay log, grounded in hand labeling.
5. Validating the detectors as formative assessments of implicit learning by comparing the performance of learners on external pre/post assessments to what the detectors suggest they understand based on their gameplay.

These methods have changed in significant ways that reduce the amount of time needed to complete all 5 steps by half without sacrificing the quality of the detectors. All Impulse detectors were created from hand labeling of videos. For Quantum Spectre, a puzzle game with static game states and a small number of solutions, we utilized interaction networks to analyze patterns in sequences of moves (Rowe et al., 2017a). We identified interaction sequences consistent with an understanding of the Law of Reflection and slope and compared those game behaviors to changes in pre-post assessments. We built a playback tool that recreates the game state sequences as players experienced them and synchronizes the labeling with gamelog and multimodal data streams. We are currently using the playback tool to develop detectors of implicit computational thinking in Zoombinis and to study the relationship between eye behaviors and implicit learning during Impulse gameplay.

We found significant relationships between Impulse and Quantum Spectre game behaviors and improvements in pre-post assessment results (Rowe et al., 2017a), suggesting the automated detectors identify gameplay behaviors consistent with physics understandings of interest. Analysis of pre-post computational thinking assessments and Zoombinis gameplay data collected during the 2017-18 school year is ongoing.

Authors

• Elizabeth Rowe, TERC

• Jodi Asbell-Clarke, TERC

• Ryan Shaun Baker, University of Pennsylvania