Predicting Learning Within a Serious Game

• In Event: 46.085 - Roundtable Session 15
  In Roundtable Session: 46.085-18 - Design Issues Regarding the Use of Games and Simulations for Learning and Assessment

Abstract

The objective of the current study is to investigate learning within a serious game called Operation ARA (Halpern et al., 2012; Millis et al., 2011). The current investigation blends Evidence Centered Design (Mislevy, Almond, & Lukas, 2003) and educational data mining methods (including WEKA and correlations, Baker & Yacef, 2009). The 10-20 hour game teaches students principles and reasoning about scientific research methodology via multiple pedagogical components, including natural language conversations, adaptive scaffolding, reading E-text, feedback, score points, generation of information, case-based reasoning, question asking, and multiple-choice questions. Students perform these activities across three training modules that teach didactic content and active application of knowledge to cases. The different training modules emphasize shallow versus deep mastery of the material (Forsyth et al., 2013). We investigated the best predictors of shallow versus deep learning across the three training modules.

College students (n = .192) interacted with Operation ARA in a pretest-interaction-posttest design. Each student’s interaction produced a log file with over 6700 variables. Following ECD, we narrowed down potential predictors that funneled into four major constructs of learning (i.e. time-on- task, generation, discrimination, and scaffolding). ANCOVAs were conducted to discover the best predictors of shallow versus deep learning after statistically adjusting for topic and prior-knowledge.

The analyses identified significant predictors for three of the four above constructs of learning. Unexpected time-on-task was not predictive of learning. There were no interactions between the predictors and depth (shallow vs. deep learning) or prior-knowledge. In the training module that taught didactic information generation and scaffolding had an incremental significant effect on learning with effect sizes of η2 = .02 and η2 = .06, respectively. Generation had a significant positive relationship with learning (r = .157) as predicted by an established theoretical frameworks (Raijmakers & Shiffrin, 1981). Scaffolding had a significant negative relationship (r = -.184), which presumably means that students who needed less scaffolding performed better. In the applied training module, both generation and discrimination had significant effects on learning with effect sizes of η2 = .029; η2= .036, respectively. Specifically, generation had a significant positive relationship with learning (r = .133) whereas discrimination had a negative relationship with learning (r (192) = -.153, p <.03). The potential trade-off between these two constructs has been discussed in the literature (Hunt & McDaniel, 1993). However, it is uncertain whether these factors are attributable to the processing activities (generation helps organize material) or individual differences (underachieving learners have poor discrimination). In the question generation module, discrimination had a significant effect with an effect size of η2 = .053 showing a positive relationship with learning (r = .175) as predicted (Raijmakers& Shiffrin, 1981).

As a final note, these findings were not discovered by data mining procedures alone, but rather through a combination of ECD and data mining. This research demonstrates the value of these technologies to structure design and assessment issues.

This research has been reviewed and approved by my university review board (IRB).

Authors

• Carol McGregor Forsyth, Educational Testing Service

• Arthur C. Graesser, The University of Memphis