Affective Learning Companions: A Serial Mash-Up of Affective Sensing, Machine Learning, and Relational Agents

• In Event: When Systems Collide: Challenges and Opportunities in Learning Technology Mash-Ups

Abstract

The Technologies. Affective sensors developed by Roz Picard’s Affective Computing Group at the MIT Media Lab and subsequently produced in a limited batch of 25 units of each sensor, included a pressure mouse, skin conductance sensors, facial expression cameras, and posture sensor chair (Burleson et al. 2004, Arroyo et. al. 2009). These sensors were connected to a java based relational agent developed at NYU by Ken Perlin that was capable of executing behavior scripts and real-time responsive interactions (e.g., based on sensor input). Advanced machine learning algorithms and user modeling were integrated as real-time classifiers capable of detecting elements of learner affect. (Kapoor et al 2007; Arroyo et al 2009; Cooper et al. 2010; Muldner et al 2010).

The Vision. Through a series of collaborations from 2004 through 2011 we sought to make use of advanced real-time affective sensing, machine learning algorithms, and adaptive affective support to advance the state of the art of Intelligent Tutoring Systems (ITS) and Affective Learning Companions (ALC). We aimed at developing ALCs and ITS that recognized and appropriately responded to learner’s affect and engagement. In addition to collaborations discussed by Greaser and D’Mello in this proposal, Burleson, Picard and Ken Perlin connected similar sensors to a dynamically expressive scriptable agent that could react in real-time to the expressions of learners, realizing, for the first time, the benefits of non-verbal mirroring in a challenging problem based setting (Burleson and Picard 2007). The sensors enabled Kapoor, Burleson, and Picard to collaborate on algorithms that could classify quitting behavior with 79% accuracy (Kapoor, et al. 2007). The outcomes, sensors and algorithms were then further advanced in collaborations between Woolf, Arroyo, Burleson, Muldner, Picard, and their research teams, to explore the nature of iterative affective feedback and the efficacy of individual sensor channels to the classification algorithms (Arroyo et al. 2009). Muldner, Vanlehn, van de Sande and Burleson then used the sensors to classify moments of excitement (Yes! Moments) (Muldner et al 2010; Muldner et al 2011). in the Andese Physics tutor. Collectively these systems and mashups aim to understand frustration and state’s of STUCK! (Burleson and Picard 2007) and advance ITS strategies that foster creativity and Flow (Burleson 2005).

The Future. We see these integrated forms of technology mash-ups contributing to increased understanding of learner’s affect and effective response strategies. They are demonstrating some of the essential elements needed to realize effective cyberlearning environments that foster learning beyond the desktop, offering support and mentorship to individual learners and collaborative teams, any-where, any time (24/7). We are creating advanced forms of personalized learning that realize two of the grand challenges of the CRA – to provide a teacher for every learner and a team of ones own (CRA 2002).

Authors

• Winslow Burleson, Arizona State University

• Ken Perlin, New York University