Research on Scaffolding: Using Algorithmic Modeling of Players' Knowledge Growth From Game Log Files

• In Event: 42.072 - Observational Methodologies Across Informal Learning Spaces

Abstract

Objectives. Digital games can provide an opportunity for players to learn new scientific knowledge (Honey & Hilton, 2011). Players’ transactions with digital games and related game performances can be automatically collected in time-stamped log files (Martin & Sherin, 2013). Like human observations however, log files are unfiltered observations made by machines of which values can only be revealed by prudent applications of appropriate research methodology. This study addresses (1) how an innovative computer algorithm called Monte-Carlo Bayesian Knowledge Tracing (MC-BKT) can capture the growth of players’ knowledge addressed by the game, and (2) how MC-BKT analysis results can be used to determine whether an embedded graphing tool worked to support knowledge growth.

Research context. Research was conducted with a game featuring a car sliding down on a ramp of which surface had some friction. See Figure 1. The ramp game consisted of five levels requiring players to apply more and more sophisticated knowledge about the ramp system. Players’ performances were scored on a 0 to 100 scale. If scored 65 points or higher, players moved to the next step within the level. If players finished all four steps within the level, they moved to the first step of the next level.

Theoretical framework. To track players’ knowledge growth, we used the Bayesian Knowledge Tracing (BKT) model (Corbett & Anderson, 1995). BKT considers a knowledge variable as latent and estimates the knowledge growth curve from observed performances as a function of time. The BKT analysis produces four parameters: initial knowledge (L0), transition (T), guessing (G), and Slip (S). See Table 1. Since conventional BKT analyses had limitations in estimating individualized parameters (Lee & Brunskill, 2012), we developed the MC-BKT algorithm to determine each player’s knowledge growth independent of the entire sample (Gweon et al., 2015).

Data sources and analysis. 42 groups of high school physics students played the ramp game where the table tool was available for all groups but the graph tool had to be activated by them. We recorded all of the log files generated from the gameplay and applied MC-BKT to estimate four BKT parameters associated with each game level for each group of players. We applied mixed effects ANOVA on each BKT parameter with player group as a random effect and graph usage as a fixed effect.

Results. After controlling for player group effects, the probability of guessing and the probability of slipping were significantly reduced for those who used graphs as compared to those who did not (Table 2). This indicates that the graphing tool reduced random errors and encouraged players to notice relationship patterns from the graph.

Significance. The MC-BKT algorithm can (1) be useful to track players’ knowledge growth in game-like simulations without utilizing external tests, (2) allow researchers to associate the effect of scaffolding tools embedded in an interactive learning environment with the knowledge growth, and (3) be combined with other data to investigate how factors such as players’ characteristics, behaviors, and task features impact the knowledge growth.

Authors

• Hee-Sun Lee, The Concord Consortium

• Sam Gweon, Physics Front LLC

• Robert F. Tinker, The Concord Consortium

• William Finzer, The Concord Consortium

• Natalya St. Clair, The Concord Consortium