Game-Based Formative Assessment for Argumentation: Mars Generation One: Argubot Academy

• In Event: 65.077 - Game-Based Assessment and Learning of Argumentation Skills

Abstract

Objectives
Affordances of games have a natural connection to principles of formative assessment. This presentation describes these connections, and provides evidence from the development, structure, and use of one game that focuses on classroom-based formative assessment for aspects of argumentation. The intended users are students and teachers in middle school.

Theoretical Framework
Leahy, Lyon, Thompson, and Wiliam (2004) published a set of five formative assessment principles. Many games contain strong analogues of these formative assessment practices, and game designers have been developing a variety of game mechanics and genres over the past 30 years that support these principles. The game we discuss draws upon a variety of game design approaches, mechanics, and genres to implement four of the five formative assessment principles within the game play, and one of the principles in the social environment surrounding the game.
The game also builds upon an existing learning progression for argumentation developed by Song, Deane, Graf, and Van Rijn (2013). It draws upon characterizations of knowledge of argumentation, organized as schemes, as described in (Walton, 1996). The game is intended to support improvements in components of argumentation in reading and writing including construction, critique, rebuttal, and knowledge of specific argument schemes such as arguing from authority, observation, and consequence.
The design also draws upon principles of learning (Hattie, 2009), specifically scaffolding to support knowledge acquisition, repeated practice with feedback to support the development of skill, and dialogue and reflection to support generalization.
Lastly, the game was developed using a process, Evidence-Centered Game Design (ECgD) that integrates aspects of assessment, learning, and game design to help meet multiple and sometimes conflicting design constraints (Mislevy, Oranje, Bauer, VonDavier, DiCerbo, Hao, Corrigan, Hoffman, DiCerbo, & John, 2014).

Methods, Data Sources, and Results
An iterative design and development process was employed including several types of increasingly large data collections (play testing, pilots, alpha, and beta testing) to evaluate and improve several dimensions of the game including usability, playability, quality of evidence, engagement, learning and feedback, and underlying statistical measurement models. Data include recording of game telemetry, cognitive interviews, survey questions, independent measures of argumentation. This iterative approach to data collection supports both top down design approaches (to determine if predictions are borne out) and more bottom-up educational data mining approaches (Mislevy, Behrens, DiCerbo, & Levy, 2012).
In the presentation, we will describe the game, underlying learning goals, and draw upon data from several different studies to make concrete the purposes of each collection with respect to the phase of design, the results of the study, conclusions, and the design decision that followed.

Significance
This project addresses several pressing research and development issues to the field of game-based formative assessment including what dimensions of games can be leveraged for learning, what learning goals can be supported, and what kinds of evidence from game-based assessment can support learning?

Authors

• Malcolm Bauer, ETS

• Tanner Jackson, Educational Testing Service

• Seth Corrigan, GlassLab Games

• Erin Hoffman, GlassLab, Institute of Play

• Maria Bertling, Harvard University

• Christopher Kitchen, Educational Testing Service

• Katherine Furgol Castellano, Educational Testing Service

• Andreas H. Oranje, Educational Testing Service

• Kristen E. Dicerbo, Pearson