Differentiating Between Unproductive and Productive Persistence in an Educational Game Using Behavioral Data

• In Event: Remote Measures of Attention and Affect: Online Data Collection in the Age of COVID-19

Abstract

Persistence in the face of challenges plays an important role in learning especially in educational games, where effort is crucial for working through increasing difficulty levels. However, recent work on wheel-spinning, where students spend too much time struggling to learn a topic, has shown that not all persistence is uniformly beneficial for learning (Beck & Gong, 2013, Kai et al., 2018). Thus, it is increasingly pertinent to differentiate between productive and unproductive persistence. This paper reports on a study aimed to better understand productive and unproductive persistence and how it relates to student progress in three puzzles of an educational game, Zoombinis. The measures of productive and unproductive persistence are used to inform adaptive scaffolding embedded within the game to meet each student’s emotional and motivational needs.

This study has two main goals: 1.) to leverage previously validated automated detectors of
computational thinking (CT) from behavioral data that will inform scaffolds to mitigate frustration and boredom within the game, and 2.) to apply educational data mining techniques to determine predictive behaviors associated with unproductive and productive persistence. The process of modeling for these two goals involves four steps: 1.) reliably hand-label Zoombinis gameplay, 2.) synchronize labels to gameplay process data, 3.) distill gameplay process data into features, 4.) build automated detectors based on human labeling.

For step 1, in previous work (Rowe et al.,2017) researchers watched Zoombinis gameplay and independently labeled gameplay efficiency related to CT, such as Learning Game Mechanic, moves indicating a lack of understanding of the game, and Acting Inconsistent with Evidence, moves contradicting evidence available from prior moves. For labels related to persistence, Not at all efficient represents an unproductive persistent state of high effort but little progress while Somewhat efficient represents a productive persistent state of steady student progress.

In steps 2 and 3, researchers computed as many as 113 features to capture CT behavior. We
previously engineered the following example categories of features in Zoombinis: 1.) Overall Gameplay, features that describe general aspects of student’s play such as outcomes of each round, and 2.) Duplicates, features that capture wasted resources. We will also create new features to represent potentially meaningful evidence of student persistence within the game.

In step 4, new detectors of persistence will be built using these human-applied labels as ground truth, trying classification algorithms from a refined set of distilled features. Detectors will be built using 4-fold student-level batch cross-validation, a process in which models are repeatedly built on 75% of the students and tested on the remaining 25%, to estimate model generalizability.
Previously cross-validated Learning Game Mechanic and Acting Inconsistent with Evidence detectors will be applied to a new sample of students with cognitively diverse needs to identify trigger points of frustration and boredom.

Success in accurately differentiating between productive and unproductive persistence and identifying trigger points for intervention can be used to infer which students are in need of digital scaffolds, thereby providing a personalized game experience that supports the affect and motivation of a broad range of learners.

Authors

• Mia Almeda, TERC

• Jodi Asbell-Clarke, TERC

• Elizabeth Rowe, TERC