The Impact of In-Game Support on Learning: A Pilot Test for Physics Playground

• In Event: 18.057 - Physics Playground, a Game for Learning: Demonstration and Posters

Abstract

Physics Playground, as a learning game, aims to foster students’ physics understanding and bolster students’ formal conceptual physics learning via gameplay. At the current stage of the project, we have developed various learning supports to assist conceptual physics learning as well as gameplay (i.e., multiple representations of the knowledge, worked examples and tips for game mechanics). In this poster, we report the results and lessons learned from the pilot study of Physics Playground aiming to investigate the effectiveness and usability of learning supports.

Forty-four eighth-grade science students (23 males and 21 females) participated in a four-day quasi-experimental study. Half of the participants were selected to play the with-support version of the game while the others played the no-support version of the game. Students were provided with 60 levels in total (i.e., 30 sketching levels and 30 manipulation levels targeting 9 physics competencies). Their conceptual physics understanding was measured after gameplay by an 18-item physics test. Participants’ experiences and attitudes were also collected through a post-study survey.

Although previous empirical experience showed playing Physics Playground can improve students’ conceptual physics understanding substantially (Shute, Ventura, & Kim, 2013), this study failed to detect any learning gains (t(43) = .96, p = .37, d = 0.14). Opposing our hypothesis, the group playing the no-support game had higher posttest scores when holding pretest scores constant (F(1,43) = 4.06, p = 0.05, d = 0.61). Furthermore, the item analysis of both pre and posttest results showed the tests were fairly difficult for the students, leading to relatively low test reliability (αpretest = 0.43 and αposttest = 0.40). A few items even showed probabilities lower than guessing and negative discrimination indices. However, based on testers’ observations and post-study survey results, students showed great satisfaction and engagement with the game experience.

The results from the pilot study provide a few potential reasons significant learning gains did not occur for students: First, the gameplay time was not long enough. Although this was a four-day study, the actual gameplay time was less than two hours due to technical and logistical issues. It is hard to achieve any substantial learning gains within this short time frame. For simplicity, the version of the game used for testing did not include the incentive system, but without incentives, students were less likely to access the physics learning supports voluntarily. Similarly, the most frequently-accessed in-game support was the solution videos. Viewing solutions helped students solve the game levels but may not aid in developing physics understanding. Finally, some students were confused with aspects of the user experience design. For example, tutorial levels were too long and easily forgotten. Once forgotten, they had to replay the tutorial levels and it broke students’ flow of gameplay.

The discussion of the results contributes to our design attributes of both in-game supports and game mechanics. The reflection on students’ learning experience also prepares us better in the next phase of the project, implementing the adaptive learning algorithm.

Authors

• Ginny Smith, Florida State University

• Zhichun Liu, Florida State University