Examining the Validity of Game-Based Intervention Effects Using Advanced Psychometric Methods and Population Data

• In Event: Teaching Informational Text With Games and Videos: A Multi-Institutional Success Story of Impact and Innovation

Abstract

Data from the Molly of Denali games are an amalgamation of data types and sources. This study combined an external measure, gameplay data, and game-based and video indicators under a single framework to examine their interplay. We also illustrate how population data can be applied to smaller subsamples. The population data are from more than 10,000 players collected "in the wild" by PBS KIDS for three months. The purpose is to capitalize on the data’s potential to provide rich and diverse information in exploring the relationship between media usage measures and children’s knowledge of informational text (IT) concepts.

We focused on the RCT data collected by EDC/SRI on Fish Camp and Sled Dog Dash. This dataset includes EDC/SRI-developed external measure items, given to both the control and treatment conditions (n = 259), as well as video measures and gameplay data for children in the treatment condition (n = 143). The psychometric properties estimated from population gameplay data (see Choi’s presentation) were applied to the RCT gameplay data.

The analyses consisted of multiple steps. First, various game-based and video usage indicators were created, among which five were chosen that represented different aspects of the online instructional experience (Table 1). Next, we conducted psychometric analyses on the external measure and gameplay data guided by three IT features (i.e., table of contents, captions, and diagrams) of the population gameplay data. Feature analysis on the external measure also identified a subset of nine items with three IT features (Table 2), and a higher order item response theory (HO-IRT) model was fit to estimate overall ability scores. In addition, we fit a diagnostic classification model (DCM) to estimate attribute profiles about mastery of the features. For the gameplay data, the same HO-IRT model and DCM were also fit using the item parameters estimated based on the population data. As such, fixed-item calibration was then used to estimate the ability scores and attribute profiles of the RCT sample. Lastly, robust hierarchical regression methods were employed with the five game-based and video usage indicators as predictor variables explaining outcome variables of the ability scores and attribute profiles.

Among a myriad of results, some noteworthy ones include the distributions of attribute mastery profiles based on the external measure and gameplay data (Table 3). While both show that most participants are likely to be classified as either having mastered none (profile 000) or all three attributes (profile 111), their results regarding the two profiles are quite opposite. Comparing the gameplay data of the EDC/SRI sample with the population data, both the percentage of profile 111 (Figure 1) and the mean of HO-IRT overall ability score (Figure 2) were higher for the EDC/SRI sample. These may be indicative of an intervention effect to the normative population. Furthermore, we found that two game-based indicators, mean total attempts per round and mean correct attempts per round, were significant for all the outcome variables used (Tables 4 and 5), pointing to their salience in predicting IT knowledge.

Authors

• Yon Soo Suh, University of California - Los Angeles

• Kilchan Choi, University of California - Los Angeles