Estimating Reliability in Game-Based Assessment: Two Illustrations

• In Event: Advances in Estimating Reliability: Innovative Approaches and Applications

Abstract

Games need to attain psychometric respectability to serve as assessment tools. Concepts of validity, reliability, and fairness carry over from traditional assessments to games, but the methods for evaluating them may not easily transfer to data typical of games. This study illustrated the application of traditional reliability estimation approaches to game-based assessment in a manner reflecting assessment context, data structure, and understanding of what was assessed. Specifically, generalizability theory and item response theory were applied to estimate reliability for measures of persistence based on data from Poptropica, a virtual world in which players explore themed “islands” with quests that players can choose to fulfill. The authors provide recommendations for retaining the concept of reliability while creating new reliability estimation routines.

Authors

• Jeffrey T. Steedle, ACT, Inc.

• Emily R. Lai, Pearson

• Paul D. Nichols, ACT, Inc.

• Kristen E. Dicerbo, Khan Academy