Use of Self-Efficacy, Effort, and Interest as Affect Metrics for Games and Simulations

• In Event: Symposium: Games and Simulations: Learning Analytics and Metrics

Abstract

Objectives/Purposes/Theoretical Framework
We view the goals, processes, and outcomes of games and simulations as having both cognitive and affective process and outcome components. We view affective outcomes of education as desirable in their own right and many are related to academic achievement. Effect sizes of .3 to .4 are reported in the literature (Hattie, 2008; Volodina, Nagy, & Köller, 2015; Rounds & Su, 2014). A similar logic is evidenced in the PISA research (e.g., Marsh et al., 2006).
One can view these affective constructs from a traits or state perspective. Traits are considered relatively enduring predispositions or characteristics of people (e.g., aptitude), whereas states are attributes of individuals’ variability over time or occasions (Spielberger, 1975). The distinction is important, as manifestations of state characteristics are highly dependent on the environment and circumstances of the specific instance, e.g., game performance.
In our research program, we measured state self-efficacy, effort, as well as trait interest. The specific format for self-efficacy and effort was a result of a crowd-sourcing study (O’Neil and Madni, 2016). We measure interest via the Job Opportunities in the Navy metric (JOIN). The U.S. Navy has embraced using interest as predictor of success, acknowledging that a recruit’s particular interests can have a bearing on their suitability to the particular Navy enlisted jobs. It has demonstrated good reliability (.88) (Farmer et al., 2003; Farmer et al., 2007). We are designing a set of Navy Life Games that will use the self-efficacy, effort, and interest metrics to judge its success.

Method/Data Sources
The context for our research has two sources of data. The research context on state self-efficacy and effort is a computer physics game called Puppybot Rescue. We revised measures of state self-efficacy and effort (O’Neil and Abedi, 1996; Blair, O’Neil, & Price, 1999). Students were randomly assigned to treatment and control conditions. Students in the control condition played another educational video game targeting unrelated physics content. The validity studies for the trait interest measure were conducted in the context of the Navy’s selection and classification program.

Results
With respect to the Puppybot Rescue game, there were significant differences favoring treatment group on performance (.23 effect size), self-efficacy (.41 effect size) and effort (.48 effect size). Metrics show fairly high reliability coefficients despite of the small number of items (5 or 6 items), i.e., alpha reliability coefficients were 0.68 and 0.80 respectively, for self-efficacy and effort. The JOIN data sources also showed high reliability and validity. These reliabilities are considered acceptable for research purposes. The set of metrics will be integrated in the Navy Life Game System.

Significance
New metrics for state self-efficacy, effort, and interest will permit additional game-based affective consequences research and also permit the evaluation of games on non-cognitive bases and will be implemented in the Navy Life Game System whose intent is to improve selection and classification of Navy recruits.

Authors

• Harold F. O'Neil, University of Southern California

• Stephen Watson