Moving Targets: Representing and Simulating Choreographies of Multimodal Pedagogical Tactics for Virtual Agent Mathematics Tutors

• In Event: 33.078 - Roundtable Session 11
  In Roundtable Session: 33.078-16 - The Learning at Hand: Gesture Production in Virtual Pedagogical Agents

Abstract

Objectives

A team of UC Berkeley Learning Scientists (Abrahamson, Director) and UC Davis Computer Scientists (Neff, Director) are collaborating on an NSF Cyberlearning EXP project, “Gesture Enhancement of Virtual Agent Mathematics Tutors.” The UCB team has been generating a library of pedagogical gestures for the UCD team to simulate virtually, based on motion capture. Our gesture-archiving effort yielded unexpected insights: (1) analysis of this library sensitized us to aspects of our own multimodal pedagogical communication, such as gaze, that we had overlooked in prior microgenetic modeling; and (2) an animated library of multimodal pedagogical tactics could serve in teachers’ professional training and development.

Theoretical Framework

Situated motion is notoriously difficult to inscribe (Guest, 1998; Kirsh, 2010). In particular, gestures in embodied interaction feature complex trajectories and morphologies not easily reducible to linear description in paper media. Despite the recent turn in the learning sciences to issues of embodiment and multimodality in teaching and learning, we still lack a precise, shared language for describing the particulars of form involved in human gesture and bodily activity. While we borrow various gesture classification schemes from other disciplines (Kendon, 2004), none offer a standardized system of description at a level of specificity necessary for three-dimensional, dynamic reproduction.

On the one hand, common gesture annotation—a cumbersome combination of verbal narratives and static images with multiple elaborate arrows—leaves critical spatio-dynamic information (e.g., trajectory) ambiguous, so that actors or teachers working from such illustrated scripts cannot accurately reenact the original movement. On the other hand, gestures in video clips drawn directly from empirical records are liable to be too low on acuity, partially obscured, insufficiently generic, and over-situated in previous virtual and physical spaces.

Methods and Results

Our solution was to circumvent inscription and reenact gestures by producing a library of animated GIFs to guide an actor’s motion capture performance (see Figure 8). To train the actor, we first built a low-tech analog of the virtual environment, using a plastic screen to stand in for the monitor screen, and filmed a tutor (VJF) acting out the entire gesture library. We then converted these clips into animated GIFs (see Figure 9).

Yet in inspecting this library, we recently realized our failure to capture naturalistic patterns of gaze that would accompany the simulated gestures (e.g., Streeck, 2009). Our analytic attention has thus unexpectedly turned to inspecting the role of gaze patterns in directing students to complex multimodal Gestalts (Mondada, 2014).

Conclusions and Scientific/Scholarly Significance

Whereas journals and online tutorials are increasingly accepting video content, quality video is difficult to generate (Derry et al., 2010) or decipher (Hall, 2000; Jordan & Henderson, 1996). Animated GIFs of reenacted gestures offer a way to ameliorate some of these difficulties. Reenacting gesture is relatively easy, clearly depictive, and circumvents IRB issues; and animated-GIF collections can be stored in broadly accessible formats (e.g., internet blogs).

Animated-GIF banks present exciting possibilities for productively disseminating gesturing techniques directly to designers of multimodal virtual pedagogical agents as well as to instructional practitioners in training.

Authors

• Virginia J Flood, University of California - Berkeley

• Alyse D Schneider, University of California - Berkeley

• Dor Abrahamson, University of California - Berkeley