A Multicomponential Analysis of Emotions During Complex Learning With an Intelligent Multiagent System

• In Event: Interdisciplinary Approaches for Analyzing Data From Multiple Affective Channels With Computer-Based Learning Environments

Abstract

Leveraging learners’ experience of adaptive (e.g., positive, neutral) emotional states has become a primary goal for ITS research. The objective is to design systems able to detect, model, and adapt to changes in learners’ emotional fluctuations in order to facilitate adaptive emotions related to effective learning. In order to accomplish these systems must: (1) detect and monitor learners’ emotions in real-time, (2) have theoretically-based rules to classify and identify fluctuations in learners’ emotional states, and (3) use pedagogical rules to provide context-relevant scaffolding depending on the detected emotions and their potential impact on learning (Azevedo et al., 2013).

This paper will discuss the results of aligning three different methods for measuring emotions that correspond to each of their components (i.e., channels) (Gross, 2013). As such, we define emotions as multi-componential (behavioral, physiological, experiential / feeling) appraisal-driven responses to objectives which have valence (positive/negative) and arousal (high/low) dimensions (Pekrun, 2011).

Data was collected from 67 undergraduate students from a North American university who interacted with MetaTutor for a 1 hour learning session (Azevedo et al., 2013, Harley et al., 2013). Our analyses focus on learners’ interactions with several PAs to learn about the human circulatory system. A webcam was used to capture videos of learners’ facial expressions which were analyzed using automatic facial recognition software (FaceReader 5.0). Learners’ physiological arousal was measured using Affectiva’s Q-Sensor 2.0 skin conductance bracelet. Learners self-reported their experience of 19 different emotional states (including basic, learner-centered, and academic achievement emotions) on five different occasions during the learning session. These emotions were measured using one item each on a 5-point Likert scale ranging from “Strongly Disagree” to “Strongly Agree.” In order to align the three methods, ten second intervals (before the self-report measure) were taken from both the Q-Sensor bracelet and FaceReader logs. Q-Sensor data was dichotomized into high and low categories using a user-dependent model where fluctuations are relative to individuals’ baseline.

Using this approach we have found a high agreement between the facial and self-report data (75.6%) when similar emotions were grouped together along theoretical dimensions and definitions (e.g., anger and frustration). This result provides evidence that facial expressions and learners’ experience of emotions are tightly coupled (Gross et al., 2011). In other words, if someone feels and expresses that they are happy, they will probably also have a matching facial expression (e.g., smile). Our recent results, examining the agreement between the Q-Sensor and these two methods, however, suggest that physiological indices of emotions do not have a tightly coupled relationship with them. Specifically, we found an agreement rate of 60.1% (κ = 0.07) between Q-Sensor and FaceReader and 45.6% (κ = .009) between Q-Sensor and the self-report measure of emotions. The highest agreement between the Q-sensor other methods was between learners’ self-reported experience of boredom and low arousal (67.6%).

Our paper expands our methodological description of how we have measured and aligned emotion data using three different methods as well as discuss the theoretical significance and applications to designing more emotionally-adaptive ITSs.

Authors

• Jason Matthew Harley, McGill University

• François Bouchet, McGill University

• Mohammed Sazzad Hussain, The University of Sydney

• Roger Azevedo, North Carolina State University

• Rafael Calvo, University of Sydney