The PBS KIDS Learning Analytics Platform: Telemetry at Scale – Best Practices and Lessons Learned

• In Event: Game-Based Measures of Learning Processes: Challenges, Methods, and Solutions

Abstract

The PBS KIDS Learning Analytics Platform (LAP) is intended to unlock learning engineering for PBS KIDS and the larger community of independent producers that create the media that PBS KIDS distributes (Roberts et al., 2016, 2019). The LAP does this by gathering telemetry data that support the study of engagement and learning outcomes. Findings enable producers and developers to optimize design towards continuous improvement of demonstrable impact at scale. As of summer 2020, approximately 100 games are LAP-enabled. The LAP safely collects detailed and anonymous data from PBS KIDS games and distribution channels that show how the media is being used, and how children interact with the playful and learning-related aspects of the media. The LAP makes it easy to collect such data sets: Data specification and collection infrastructure are seamlessly integrated into PBS KIDS distribution channels and products. The LAP stores data in a manner permitting flexible and efficient ad-hoc analysis, and the data are stored economically for as long as useful questions remain. Analysts provide services to content managers and game developers to answer questions about how design approaches translate into real-world usage and impact. Current platform development activity is laying the research and software infrastructure groundwork for conducting fast randomized controlled trials at scale. This talk will review lessons learned and best practices about good telemetry design. Topics will include (a) data specification and examples; (b) future-proofing against research questions that evolve over time, while remaining cost effective; (c) the value of reporting facts and not interpretations; (d) capturing user inputs and responses, system inputs and responses, and game / puzzle / simulation states; (e) supporting use in offline mode; (f) tools for researchers and integration of analytics within recruited studies; (g) promoting downstream analysis efficiencies with smart data specifications; and (h) an example of a typical analysis of LAP data.

Author

• Jeremy Roberts, Public Broadcasting Service