Identifying Frustration Moments Using Facial Emotion Recognition and Behavioral Data Mining

• In Event: Remote Measures of Attention and Affect: Online Data Collection in the Age of COVID-19

Abstract

Frustration is a natural part of the learning and problem-solving process. As a sign of engagement, frustration can be part of a desirable and rewarding experience in which a learner or problem-solver iteratively tries (and fails) to overcome until ultimately triumphing over challenges. However, there is the risk of rage-quitting from the task with reduced motivation when frustration persists, potentially due to the imbalance or mismatch between the demand or structure of the learning environment and a learner’s zone of proximal development. Identifying whether and the moment when the state of frustration becomes demotivating or disengaging, therefore, is important for the design of an adaptive learning environment that will dynamically choose a course of action—for example, adjusting the task challenge or providing timely learner support.

In this study, we explored the feasibility of using facial emotion recognition along with behavioral data mining to identify disengaging frustration moments during digital game-based learning. With video recordings of middle-school students’ play of an educational puzzle game—Zoombinis. In Phase 1, we implemented two open-source facial-expression recognition software (i.e., Emotion-Recognition using Python and OpenFace) to capture students’ basic emotional states (e.g., angry, disgust, sad, and scared) with cross-validation. We also conducted a qualitative behavior analysis with the recorded participants’ actions and reactions during gameplay. Based on the outputs of two alternative facial emotion-recognition software and the qualitative behavior analysis, a data-driven conjecture on the specific emotions and their related action units that signify the state of frustration was made.

In Phase 2, we created the time series graph of the captured emotional states across participants, as shown by Figure 1. To detect gameplay moments that indicate negative arousal or emotions (and potentially disengaging frustration), we used the nearest rank method (Johnson & Kuby, 2007). Specifically, we computed the percentile score for each emotion (and related facial action units) time series and extracted the gameplay periods (or moments) that contain high peaks—80th+ percentile on a time series. These extracted, candidate frustration periods were further corroborated by qualitative behavior analysis and human judgement.

In Phase 3, we examined participants’ Zoombinis gameplay logs and delineated their logged gameplay actions during the candidate frustration periods. We then conducted associated rule mining with these logged gaming actions using Orange—a data-mining toolkit (Demšar et al, 2013) and Apriori algorithm (Agarwal & Srikant, 1994). For the rule mining, the cut-off event-transition probability value was 0.3 (indicating 3% chance of co-occurrence), and the Lift measure should be higher than 1 with Leverage being not equal to zero. The extracted event transitions were then sequenced chronically.

In Phase 4, through a comparative analysis with the salient gameplay event transitions and the players’ task performance across the candidate frustration periods, we identified two clusters of gameplay event transitions, with one resulting in task completion and another yielding a quitting behavior. These gameplay event transitions will then act as behavioral indicators of either motivating or disengaging state of frustration. In summary, this four-phase approach, using both physiological and behavioral indicators, contributes a cross-validated detection of frustration.

Authors

• Fengfeng Ke, Florida State University

• Jewoong Moon, Florida State University