Application of Advanced Psychometric Models to Population Game Play Data From Molly of Denali Games

• In Event: Teaching Informational Text With Games and Videos: A Multi-Institutional Success Story of Impact and Innovation

Abstract

As a part of the RTL program, we have for the first time used population data to generate insights about players' behavior and IT competency. By population data we mean gameplay data from children using the Molly of Denali game “in the wild” under natural conditions and not part of any research. To date, our previous studies have used a sample of children that have been recruited specifically for a particular study. In contrast, population data, by definition, reflect all the children who played the game.

The data for this study were from the Molly of Denali games whose curriculum is designed for children ages 4-8 and is developed to target informational text (IT) standards from PBS KIDS Literacy-English Language Arts (ELA) Learning Framework Version (PBS KIDS, 2016). The population data from two particular games, Fish Camp (FC) and Sled Dog Dash (SDD), were collected by PBS KIDS in a 3-month window, from January to March, 2020. The sizes of data are, respectively, 2.1 gigabytes with 10,963 unique players and 9.8 gigabytes with 15,953 unique players.

The psychometric analysis of the population data revealed several interesting findings. In both games, the multiple-choice-like prompts and responses around the informational text content were conceived as test items. First, there was a very small percentage of players completing all the items presented in the games. For FC, only 1.9% of players completed all 12 items and 11.0% of players completed at least six items. For SDD, only 0.4% of players completed all 54 items; 4.1% and 19.8% of players completed at least two thirds (36 items) or one third (18 items), respectively. The analyses showed that a very high percentage of players reached the correct answer on their first attempt. The mean correct response for each item was greater than .6 for the 12 items in FC and greater than .6 for 52 of 54 items in SDD. These results indicate that items in both games were quite easy for the players in the subpopulation.

By conducting a feature analysis of the items, we identified three informational text features (i.e., table of contents, captions, and diagrams) and mapped each item to a corresponding feature. The results of fitting a two-parameter logistic multidimensional item response theory model show that the three factors are moderately to highly correlated with each other. In addition, we examined the population’s mastery level of the informational text attributes by fitting a higher order diagnostic classification model. The estimated attribute mastery profile for FC data found that 54.7% of the players in the subpopulation (n = 1,205) mastered all three attributes, while 32.0% of the players mastered none of the attributes. For SDD, 52.7% of the players in the subpopulation (n = 3,158) mastered all three attributes, while 35.7% of the players mastered none of the attributes. In addition, about 11.5% of the FC players and 6.8% of SDD players mastered only one attribute.

Authors

• Kilchan Choi, University of California - Los Angeles

• Tianying Feng, National Center for Research on Evaluation, Standards, and Student Testing

• Charles Parks, University of California - Los Angeles

• Elizabeth Redman, University of California - Los Angeles

• Gregory K.W.K. Chung, University of California - Los Angeles