Use of Crowdsourcing as a Formative Evaluation Technique for Game Usability Studies

• In Event: Game Design Issues for Learning and Assessment

Abstract

Why games? Computer-based games are becoming increasingly prevalent in both education and training. Games can improve the quality of education and training by providing information adapted to the needs and abilities of the individual. Game technologies have already been embraced by most students.
This presentation suggests how education and training games can be developed that are scientifically-based and outcomes-driven. One critical tool is how to efficiently conduct formative evaluations in context of usability studies. Formative evaluation is a systematic collection of information that takes place before or during a project or product implementation with the goal of improving design and performance (Flagg, 2013). Recent research has also provided evidence for the utility of formative evaluation in strengthening intervention outcomes and implementation (Brown & Kiernan, 2001; Flagg, 2013). One of the most important steps in the game development process is determining what features and mechanics best represent the instructional and pedagogical sequence which would be essential to student learning. A common way of collecting information to fulfill this formative evaluation process involves performing usability studies, thinkaloud sessions, and testing with the target population (Adams, 2013; Pinelle et al., 2008). In contrast, we used crowd-sourcing. Crowd-sourcing is emerging as a novel process to obtain feedback (Stolee & Elbaum, 2010; Zhao & Zhu, 2012). The basic idea behind crowdsourcing is to leverage a global community of users with different talents and backgrounds to help perform a task, such as solving a problem, classifying data, refining a product, and similar to formative evaluation, gathering feedback (Parvanta, Roth, & Keller, 2013; Stolee & Elbaum, 2010; Zhao & Zhu, 2012). Participants were obtained through Mechanical Turk, which is a crowdsourcing web service associated with Amazon (Buhrmester, Kwang, & Gosling, 2011). The researchers implemented crowdsourcing as a formative evaluation tool to determine the usability of four icons for teaching antibully strategies for elementary school children in the context of a physics game of the four icons tested based on crowd-sourcing data three can be used “as is” but the remaining icon would need to be revised.

Authors

• Harold F. O'Neil, University of Southern California

• Ayesha Madni, University of California - Los Angeles