Secondary Analysis of the Cat in the Hat Knows A Lot About That RCT Gameplay Data: Linking Gameplay Indicators and Outcome Measures

• In Event: Digital Media’s Support of Early Science Learning: Three Research Perspectives on a Popular Kids Program

Abstract

Introduction
Children’s moment-to-moment behavior in digital learning media (i.e., their telemetry) can provide rich information of their media use. If the telemetry can be linked to learning outcomes, then how children’s use of the media resources relates to their learning outcomes can be examined.

Research Questions (RQ)
To what extent does: (a) children’s gameplay and media usage relate to their learning outcomes? (b) how children accessed the resources relate to their learning outcomes?

Study Population
Data were from 219 4-5yo children in the first study presented in this panel and represented children’s interaction with three games that had telemetry and 20 videos that were associated with those games. This study is discussed in another paper in this symposium.

Method
The outcome measures were science achievement (Lens, Greenfield, 2015) and science and engineering practices (as represented by three hands-on performance tasks [PT]).
Five indicators were derived from the telemetry: number of levels completed, number of repeated levels, overall accuracy, mean interaction level, and number of videos watched. These indicators were used to relate gameplay behavior and video access to outcome measures.

In addition, indicators representing the transitions of games or videos to other games or videos were derived from the telemetry to relate transitions to outcome measures.

Results
RQ1: Robust hierarchical regression was used to separately model the relationship between the Lens and PT scores and the gameplay and video indicators, controlling for age, sex, and prior knowledge. The results show that overall accuracy of children's gameplay, across all three games, was related to performance on the Lens (r = .31***) and PTs (r = .26***). Performance on the PT was also related to the number of levels in a game repeated (r = .22*) and the times a video was watched (r = .16*).

RQ2: Figure 1 shows the transitions among the different games and videos for high and low performers based on a median split of the posttest Lens scores. The high-performing group seems to have more transitions between assets of the same concept, especially transitions from videos to the game targeting the same concept.

Multiple regression (OLS) was used to model the relationship between the types of transitions and the PT outcome controlling for age and sex. Three transitions were found to be significant: (a) video to game of the same concept (β = .18**); (b) game to game of the same concept (β = .25**); and (c) game to game between different concepts (β = -.33***). Staying within a game and watching a video and then playing a game of the same concept are positively associated with higher performance on the PT. Conversely, playing a game of one concept and then playing a game of another concept is associated with lower PT performance.

Implications include (a) telemetry can be used to better understand how children interact with media and identify the resources related with learning outcomes; and (b) more research is needed on how children’s media use relates to learning, which may help design more effective interventions.

Authors

• Elizabeth Redman, University of California - Los Angeles
  Presenting Author

• Gregory K.W.K. Chung, University of California - Los Angeles
  Non-Presenting Author

• Yon Soo Suh, CRESST/UCLA
  Non-Presenting Author

• Tianying Feng, CRESST/UCLA
  Non-Presenting Author

• Charles B. Parks, CRESST/UCLA
  Non-Presenting Author

• Kilchan Choi, CRESST/UCLA
  Non-Presenting Author

• Jeremy Roberts, Public Broadcasting Service
  Non-Presenting Author

• Naomi Hupert, Education Development Center
  Non-Presenting Author