Application of Advanced Psychometric Models to Population Game Play Data From Arthur Games

• In Event: Designing and Validating Games to Measure Social and Emotional Learning: Validation Results and Lessons Learned

Abstract

Objective
This study aims to understand how children interact with Arthur games in the population, and to explore a psychometric modeling approach that estimates a player’s social and emotional learning (SEL) performance.

Background
UNIVERSITY1's prior work on games and measurement suggests that games can be used for measurement purposes (e.g., CHAIR et al., 2016; AUTHOR3 et al., 2021; AUTHOR4 et al., 2018). This study extends the findings of the recent UNIVERSITY1 SEL Games Validity Study (AUTHOR4 et al., 2021) to identify gameplay patterns and SEL subconstructs (e.g., self-awareness, social awareness, self-management) and apply game-based indicators (GBIs) to psychometric models to measure latent gameplay proficiency at the population level.

Methods and Data Sources
The population data from players were gathered by the PUBMEDIAORG1’s Learning Analytics Platform (LAP) for three Arthur games. By “population” we mean players who played the games naturally without any intervention by the research team or PUBMEDIAORG1. The data collection window was June 24 to July 26, 2021. The sample size for each game was 5,000 players who had completed all the levels in the game, and an additional random sample of 5,000 players who did not have any constraints in completion of levels.

Results
For Tower of Cows (subconstruct: persistence), three GBIs appear to reflect the amount of effort spent on playing Tower of Cows, especially after failure (i.e., time spent, number of attempts, number of reattempts). One GBI appeared to capture players’ skill at playing Tower of Cows (i.e., the number of first correct attempts). We also identified GBIs related to players’ propensity to attempt hard versions of items. A Rasch model was used to model correct first attempt data and to model propensity to choose the hard version of items. A Rasch Poisson counts model was also used to model reattempt data. Results suggest that gameplay reflects an ability component (i.e., skill at playing the game) and a persistence component (i.e., continuing to play the game after failure in the game) that are confounded.

For Muffy’s Party Planner (subconstruct: social awareness), three GBIs were used: reaction score, positive valence score, and correct score. An item response tree model was used to model player’s decision process. Results suggest three response processes related to players’ selection of party options: (a) selections that resulted in a positive valence response from the game character; (b) players’ propensity to choose options that resulted in a negative response from the game character; and (c) players’ propensity to choose options that resulted in the most positive response from the game character.

For Kindergarten Helpers (subconstruct: social awareness), two GBIs (number of correct first attempts, number of misses until correct) were used to measure social awareness. A Rasch model was used to model the correct first attempt data and a Rasch Poisson counts model was used to model the number of misses until correct. Results suggest that the items were too easy; thus, not much information can be gleaned from gameplay regarding a player’s SEL skills.

Authors

• Yon Soo Suh, NWEA
  Presenting Author

• Kilchan Choi, University of California - Los Angeles
  Presenting Author

• Tianying Feng, University of California - Los Angeles
  Non-Presenting Author

• Charles Parks, University of California - Los Angeles
  Non-Presenting Author

• Gregory K.W.K. Chung, University of California - Los Angeles
  Non-Presenting Author

• Elizabeth Redman, University of California - Los Angeles
  Non-Presenting Author