Use of Feature Analysis as Game Design and Evaluation Tool

• In Event: Game Design Issues for Learning and Assessment

Abstract

The objectives of this presentation are to describe the process of feature analysis, and illustrate its use in the design, improvement, and evaluation of game interventions. The larger purpose of this work is to create a set of validated features for use by developers and assessment designers. In our studies of technology-development and evaluation, we have undertaken the qualitative analysis of features to determine those that should be included in a technology implementation or assessment to support effectiveness or validity inferences. The features can be used either as specifications or in post hoc analyses of instruction or assessment, linked to diagnostic psychometric models. The features have been drawn from ontologies of content standards (Iseli, 2012) and cognitive demands. This approach is intended to assure the validity of instructional design and assessments. The approach is being generalized across different environments. For example, in a state assessment over on a three-year period and involving four grade levels and both mathematics and English language arts. The results indicated that associated features of content, cognition and task accounted for significant variation (>.50).
These features can be tagged in databases so to inform their contribution to learning, both in terms of variance accounted for and their relationship to types and level of learning achieved. Although applied in many of our studies, the design of physics games for children is the current case. The goals for the game were selected from the Next Generation Science Standards regarding physics topics. An ontology of sub-ideas related to the conceptual understanding of physics content, e.g., friction, was developed. In addition, a problem-solving ontology was also used as a source of design features for the game and for assessments. Task features, involving numbers of scenarios, stimulus properties (how many concepts per frame and range of response modes) were developed. Together these were used to guide the development of game levels and to assure that outcome measures represented the range and difficulty of content, problem solving, and tasks expected to be learned, moderated by the game mechanic in use. In the development of game outcome measures, the use of features unexpectedly allowed the rapid development of performance assessments that met content validity requirements implied in the standards and in game specifications. The administration of the game to elementary school children resulted in student learning as indicated by a moderate effect size.

Author

• Eva L. Baker, University of California - Los Angeles