Predicting Learning in a Serious Game With Multiple Pedagogical Components

• In Event: Game Design Issues for Learning and Assessment

Abstract

What is the impact of multiple pedagogical components on learning within serious games? The specific serious game we investigated was called Operation ARA (Halpern et al., 2012; Millis et al., 2011). We blended Evidence Centered Design (Mislevy, Almond, & Lukas, 2003) and educational data mining methods (Baker & Yacef, 2009) to assess what predicts learning in the game. The 6-12 hour game teaches students principles and reasoning about scientific research methodology via multiple pedagogical components, including natural language conversations, adaptive scaffolding, reading E-text, feedback, score points, generation of information, case-based reasoning, question asking, and multiple-choice questions. Students perform these activities across three training modules that teach didactic content and active application of knowledge to cases. The different training modules targeted both shallow and deep knowledge of the material. We investigated the best predictors of shallow versus deep learning across the three training modules.
College students interacted with Operation ARA in a pretest-interaction- posttest design. Each student’s interaction produced a log file with over 6700 variables. Following ECD, we narrowed down potential predictors that funneled into four major constructs of learning: time-on-task, generation of ideas, discrimination, and amount of computer scaffolding needed by the student. ANCOVAs were conducted to discover the best predictors of shallow versus deep learning after statistically adjusting for topic and prior-knowledge. The analyses identified significant predictors for three of the four above constructs of learning. Unexpectedly, time-on-task was not predictive of learning. There were no interactions between the predictors and depth (shallow vs. deep learning) or prior-knowledge. In the training module that emphasized factual information, generation of ideas was significant positive predictor and need for scaffolding was a significant negative predictor of learning over and above other components (partial η2 = .02; partial η2 = .06, respectively). In the applied training experience, both generation of ideas and discrimination were significant predictors of learning (partial η2 = .029; partial η2= .036, respectively). As expected, generation of ideas was a positive predictor, but unexpectedly discrimination was a negative predictor, for reasons not fully understood. Perhaps the discriminations in training were too subtle or perhaps there was a trade-off between generation and discrimination and generation dominated. In the question generation module, discrimination was a significantly positive predictor of learning (partial η2 = .053).
This research demonstrates the value of these technologies to structure design and assessment issues. Moreover, the impact of any one pedagogical component is modest when it is included among an ensemble of other components. The impact of one component may be robust when playing alone, but modest when joined with the rest of the ensemble. These findings were not discovered by data mining procedures alone, but rather through a combination of ECD and data mining. This process is an effective method of uncovering important predictors in game environments based on theoretical principles.

Authors

• Carol McGregor Forsyth, Educational Testing Service

• Arthur C. Graesser, The University of Memphis