10 Years After: Shifts and New Challenges in Game-Based Assessment

• In Event: 43.015 - Game-Based Assessment: How Has the Field Matured Over the Past 10 Years?

Abstract

In this work, we discuss major shifts in game-based assessment (GBA) research driven by the progress made in the last 10 years. We illustrate each of these positive shifts, drawing upon our current work on microgames, games with one or two mechanics that allow for ease of versioning and experimentation, and extended games, games that provide rich experiences for players to engage in sequential decision-making over time.

We view these four shifts as natural advancements as the field begins to mature and illustrate each of these shifts and associated challenges with examples from previous work. These shifts represent a field transitioning from making an existential case to addressing foundational questions in a systematic way.
1. From gamification and homogeneous design approaches to synthesis of assessment, learning, and game design processes. Early work often employed a single design approach (e.g. applying assessment design to game or game design to learning materials). As the field has matured and multi-disciplinary research blossomed, synthetic approaches that fuse design practices across fields for specific purposes have emerged (e.g. evidence-centered game design (ECgD), an approach used for several of our projects which explicitly combines representations game designers use such as a Macro design, with representations of assessment designers such as a Q-matrix).
2. From the use of outcome data and external data to sophisticated use of in-game process data for making inferences about what players know and can do. Early work often used end-of-game results (e.g. final score) or embedded assessment items. As the field has matured, researchers have begun to make better use of the sequence of choices that game players make as evidence of different strategies taken, concepts understood, and levels of expertise with respect to constellations of competencies. To make inferences from this sequential decision-making data, new approaches are being employed, such as partially-observable Markov decision processes (Lamar, 2018), that draw upon different fields such as machine learning, economics, statistics, and psychometrics.
3. From arguments that games are motivating to the articulation of motivation-related constructs and models of motivation in learning and game play. Researchers are less concerned about the potential for games in general and have become more focused on understanding why players become more/less engaged, under what conditions certain outcomes apply, and potential theories for explaining, designing, and generalizing desired effects (e.g., self-determination theory, expectancy value theory; Jackson & McNamara, 2017).
4. From the creation and study of single games to the creation and study of ecosystems of games (shift in scale and context). Game interactions don’t happen in isolation (especially if the games are repeated). As more contexts of use are considered during development, designers have started to create more platform environments to leverage that context and integrate experiences (i.e., platforms surrounding individual games and tasks). Researchers still struggle to disentangle complex order and attempt effects, but designs are improving to help capture dynamic changes and impacts of context over time (Snow, Jackson, & McNamara, 2014).

Authors

• Malcolm Bauer, Educational Testing Service

• Tanner Jackson, Educational Testing Service