Uncovering Learner Clusters and Optimal Content in Digital Gamified Interventions With Machine Learning

• In Event: Infectious Learning and Teaching: Opportunities and Challenges of Integrating COVID-19 Into K–16 Education

Abstract

OBJECTIVE. Apply machine learning to identify clusters of learners and optimally effective content from a large dataset of interactions with digital gamified educational intervention on vaccines.

FRAMEWORK. Compounding public health challenges posed by the COVID-19 pandemic is an infodemic of misinformation regarding risks, prevention, and treatments (WHO, 2020). However, several prior attempts to refute misconceptions about topics like COVID-19 have been inconsistent (Trevors & Duffy, 2020), in part because of a flawed assumption that misconceptions originate from a simple deficit of information rather than a complex interaction of psychosocial factors that vary across subgroups (Loomba et al., 2021). As such, a one-size-fits-all approach in public education efforts is unlikely to attain broad success. This study leveraged unique strengths of cognitive research on belief change, design principles of gamification, and machine learning to analyze existing large datasets of digital gamified intervention to uncover unique learner profiles and optimally effective content to inform the design of future interventions.

METHOD & DATA. In 2020, we developed and evaluated three digital game interventions to correct misinformation related to COVID-19 and vaccines in Canada and the US (Ntotal=221,779; Figure 2). Games presented 18-24 content pages that presented one true or false claim about COVID-19. Participants swiped left or right to indicate their agreement or disagreement with the claim. For incorrect responses, the next screen presented immediate feedback that followed an evidence-based design for belief revision (Kendeou et al., 2014). Participants then reported their emotional reaction to the corrective content (i.e., happy, angry, anxious, or skeptical). The interventions used game elements and mechanics (e.g., challenge, visible progress). Last, participants reported support for public health policies (e.g., mask wearing) and intent to receive a COVID-19 vaccine.

FINDINGS. Preliminary cluster analysis was performed on correct/incorrect responses to 17 knowledge claims from an intervention focused on vaccines. A three-cluster solution produced adequately differentiated, non-redundant profiles with sufficient number of cases in each (>5%; Figure 3). Participants with the lowest level of vaccine knowledge attained 47% correct (50% chance level) and were the smallest cluster; a middle level attained 86% correct and was the largest cluster; the highest level scored 96% correct and were 44% of the sample. As evidence of solution validity, all clusters significantly differed in their intent to receive a vaccine (p < .0001, η2 = .139; highest cluster = 4.6/5, lowest cluster = 2.8/5) and the lowest cluster significantly underperformed on a post-game test of learning after controlling for prior knowledge (ps < .001). Replicability of cluster solutions and optimally effective content for each cluster will be further investigated via additional unsupervised and reinforcement learning analyses.

SIGNIFICANCE. Given differences in specific needs and concerns across subgroups of the population, a one-size-fits-all approach to public education campaigns will likely reduce its effectiveness. By identifying unique clusters of learners from large datasets, digital interventions can be optimized by adapting instructional strategies to learners’ strengths and needs. By shedding light on distinct learner profiles, the current study represents a needed step towards augmenting digital learning interventions.

Authors

• Greg Trevors, University of South Carolina

• Pooyan Jamshidi Dermani

• Jan Ostermann

• Sayward Harrison