Detecting Signal From Noise While Protecting Data Privacy

• In Event: Interrogating the Truth While Protecting Children's Privacy

Abstract

PUBMEDIAORG1 has developed a Learning Analytics Platform (LAP) to extract data-driven insights with the intention of understanding how to measure and promote learning impact, while at the same time preserving the safety and privacy of children. For this reason, PUBMEDIAORG1 intentionally makes choices that result in the gameplay telemetry we collect from our publicly facing games being completely anonymous. This, in turn, increases the challenges in extracting value from that data. This talk will share how PUBMEDIAORG1 studies the anonymous data, and several strategies we employ to safely augment its utility, including how PUBMEDIAORG1 partners with UNIVERSITY1 to conduct IRB-regulated recruited studies designed to amplify the insights that can be gained from population telemetry.

While there are sophisticated methods for accurately authenticating individuals, they present usability barriers and/or public trust issues. To promote a safe and engaging experience, PUBMEDIAORG1 has opted in its development of LAP to prioritize security and safety over identification accuracy. This is accomplished using hashed identifiers (generated e.g. upon app installation) which do not encode any information about the user or their device, and cannot be transferred across devices or properties (sites). Furthermore, no potentially sensitive information is ever associated with individual user LAP gameplay data (e.g., user agent, zip code, etc). These measures significantly reduce the risk of user re-identification.

However, using anonymous data poses significant challenges to analysis. This talk will address some of these issues and how we mitigate them. For example, without demographic or background information about users, there is little opportunity to do subsample analyses. To address this limitation, PUBMEDIAORG1 is evaluating the development of innovative analytics systems that compute aggregate geospatial and device-level metrics, while discarding all links to the already anonymous individual-level data.

Further, a fruitful strategy for augmenting anonymous data has been to partner with UNIVERSITY1 to conduct IRB-regulated recruited studies which allow the collection of additional, detailed participant data. Combining recruited studies analysis with population data analysis, UNIVERSITY1 is able to replicate methods and analyses of recruited study data and establish that game-based indicators can be generalized to population data.

This talk will discuss how, although using anonymous data presents challenges, large N anonymous data has allowed PUBMEDIAORG1 to:
-Gain insights on how kids interact with educational media in naturalistic settings.
-Construct psychometric models to empirically quantify both a game’s challenge level and a player’s skill level.
-Develop game-based indicators related to learning outcomes, and understand whether educational media leads to demonstrable change in learning goals.

Authors

• Cosimo Felline, Public Broadcasting Service
  Presenting Author

• Jessica Younger, PBS
  Non-Presenting Author

• Kelly Corrado, Public Broadcasting Service
  Non-Presenting Author

• Jeremy Roberts, Public Broadcasting Service
  Non-Presenting Author

• Kilchan Choi, University of California - Los Angeles
  Non-Presenting Author

• Elizabeth Redman, University of California - Los Angeles
  Non-Presenting Author

• Gregory K.W.K. Chung, University of California - Los Angeles
  Non-Presenting Author