Why Measure Social and Emotional Learning Using PUBMEDIAORG1 Games for Ages 4–6?

• In Event: Designing and Validating Games to Measure Social and Emotional Learning: Validation Results and Lessons Learned

Abstract

This talk will lay out the rationale and high-level approach taken when PUBMEDIAORG1set out to commission, distribute, and study three new digital games for 4-6 year-olds designed to measure skill around social and emotional learning (SEL) concepts.

PUBMEDIAORG1 enjoys a large audience, reaching over 4.5M kids on TV, 17.7M kids visit PUBMEDIAORG1-owned digital properties, and 5.4M kids watch PUBMEDIAORG1-owned properties on YouTube annually. PUBMEDIAORG1 has a massive and consistent pipeline for creating educational games to add to our growing library. We currently have hundreds of existing games, with over 20 new games launching each year. PUBMEDIAORG1 also controls multiple high-profile large-scale game distribution platforms, such as the PUBMEDIAORG1 website (PUBMEDIAORG1_WEBSITE, 351 games), and the PUBMEDIAORG1 Games app (220 games), and in 2021 tracked robust, safe, and anonymous learning analytics for approximately 104M game sessions.

PUBMEDIAORG1 is very interested in both promoting and demonstrating learning impact at scale. We conduct formative studies to catch glaring issues during production. We conduct analytics studies to learn how our media is being used at scale. Additionally, we employ summative studies to demonstrate impact that allows us to generalize to the PUBMEDIAORG1 public audience. Given our analytics capabilities, we wondered if we could measure impact directly from the public audience? In order to do so, we would need to start with validated game-based indicators of skill and mastery.

PUBMEDIAORG1 games are interactive surfaces that (depending on design) can allow kids to get exposure on, explore, and practice content and challenges related to the game's learning goals. Social and emotional learning is one of the key educational priorities for PUBMEDIAORG1. Could games be designed with game mechanics that measure skill and mastery in the SEL domain? If so, then PUBMEDIAORG1 would have a new kind of tool in the toolbox to help directly measure the impact of the various forms of media produced and distributed at large scale.

We will discuss the process to accomplish this project including (a) expanding data collection capabilities to get the gameplay data needed; (b) commissioning world-class game designers and developers to create three new games; (c) distributing the games to the large-scale PUBMEDIAORG1 Games app audience; and (d) engaging top-notch researchers to develop game-based measures and psychometric models from the gameplay data and to conduct research to explore validity.

The implications for this work are profound and will be presented and explored. With a library that is always growing, the potential exists to slowly add to and refine measurement capabilities (e.g., to new learning goals) over time. If PUBMEDIAORG1 can directly measure SEL skill level and mastery at a point in time, then future time-evolution models may efficiently and directly measure SEL learning at scale. With PUBMEDIAORG1’s reach, future work may be able to provide a nationwide snapshot of how kids are doing on the SEL topics measured via the games.

Authors

• Jeremy Roberts, Public Broadcasting Service
  Presenting Author

• Silvia Lovato, PBS KIDS Digital
  Non-Presenting Author

• Kelly Corrado, Public Broadcasting Service
  Non-Presenting Author

• Cosimo Felline, Public Broadcasting Service
  Non-Presenting Author