Poster #145 - Towards Measuring Skill Development Using Large-Scale Anonymous PBS KIDS Gameplay Data

• In Event: Poster Session 01

Abstract

Can PBS KIDS measure learning and skill development directly in anonymous children from the public audience via games? Establishing engaging and unobtrusive assessments in naturalistic settings holds the potential to obtain high-quality information about skill and mastery from motivated participants without the stress of a conventional test, and while promoting joy. This poster will present PBS KIDS and UCLA CRESST research into how anonymous gameplay data from the PBS KIDS Games app can be used to measure skill and whether gameplay patterns can reveal changes in skill over time.

We propose to analyze data from seven educational media games from which we collected data over the course of 5 months to conduct the following activities. The team has identified games with game mechanics theorized to support measurement of specific skills. To model learning over time, the team will (a) perform psychometric analysis of game levels, (b) use multidimensional item response theory and diagnostic classification models to estimate players’ point-in-time skill levels and mastery on game learning goals, and (c) model change across the skill and mastery measurements. To provide preliminary evidence for or against validity of models constructed using only anonymous gameplay data, we will compare results across groups of games with design features rated as more or less likely to produce behavioral changes related to the learning goal.

Challenges and innovations:

The application of methodologies developed for formal assessments to naturalistic gameplay may violate assumptions of the models. We will systematically test these assumptions with the large N afforded by our platform; millions of kids are represented in the data set. However, new forms of modeling may be required to achieve our goals.

The number of “good items” encountered by “enough” individuals may not be sufficient given a) the packaging of the educational media being studied promotes engagement with a large number of games - presumably reducing return rates to individual games; and b) measurement was an explicit design goal for only a subset of the included games. We are examining a range of game mechanics (forced-choice vs free response) and game designs (level-gating vs free-choice of levels) to understand which designs are most suitable for measuring skill.

Confidence intervals of skill measurements may be wide, limiting the meaningfulness of the learning trajectories across them. We are exploring the timescales over which estimates are most reliable/useful.

The implications for success are profound. Given the reach of large educational media companies such as PBS KIDS, future work may be able to provide a nationwide naturalistic “snapshot” of what kids are learning as measured via the games. Success could also help establish the foundation of a system for using a portfolio of validated game-based measures to evaluate other media components - for example newly launched videos and games. Finally, assessments conducted unobtrusively in naturalistic settings could enable systems that react to promote learning.

Authors

• Jeremy Roberts, Public Broadcasting Service
  Presenting Author

• Elizabeth Redman, University of California - Los Angeles

• Jessica Younger, PBS KIDS

• Cosimo Felline, PBS KIDS

• Gregory K.W.K. Chung, University of California - Los Angeles

• Tianying Feng, CRESST/UCLA

• Charles B. Parks, CRESST/UCLA

• Kilchan Choi, CRESST/UCLA

• Kelly Nicole Corrado, Public Broadcasting Service