Challenges in Scaling Game-Based Assessment

• In Event: 43.015 - Game-Based Assessment: How Has the Field Matured Over the Past 10 Years?

Abstract

Over the past decade we have seen many examples of individual game-based assessment efforts that sought to use principled design methods to maintain the psychometric soundness of the related assessments. Unfortunately, we have not seen these efforts realized at scale. This poster will explore the challenges faced in bringing game-based assessment into schools across the U.S. and globally using examples from multiple efforts to do so.

Essentially, the time and budget required to make good game-based assessments for an entire curriculum exceeds the time and budget available do so. Task design in game-based assessment requires collaboration between game designers, assessment designers, and content experts in ways that they are not used to collaborating. This collaboration takes time. In addition, game-based assessment presents significantly more challenge in evidence identification than traditional forms of assessment. Understanding which actions in which contexts provide information about constructs of interest takes significantly more time and experimentation/iteration than evidence identification in traditional forms of assessment. Finally, sophisticated evidence aggregation models, such as Bayesian Networks, Markov Models, and Diagnostic Classification Models require significant time and expertise to build, and require software systems that do not currently exist at scale.

The use of games in classrooms requires integration into the other elements of instruction. A single game covering a narrow slice of the curriculum will not result in this kind of integration nor the kind of change in assessment practice that proponents want. Finally, the market demand is not there to support this work. While many surveys show that teachers are interested in games and formative assessment, and many use games in their classrooms, they are not referring to the psychometrically sound games described here. Experience in one of the largest education providers in the world indicates teachers and schools cannot or will not pay much more for a curriculum with game-based assessment than one without it. This leads to the result that, in the current state, game-based assessment is not scalable.

The paper finishes by exploring the pros and cons of various ways to address these scalability issues, including: 1) focus on simulations and use of playful elements, 2) simplification of statistical models, 3) use of artificial intelligence and machine learning in evidence identification, 4) use of templates, and 5) new business models.

Author

• Kristen E. Dicerbo, Pearson