Effective Instructional Gestures: Design Principles for Virtual Pedagogical Agents

• In Event: 33.078 - Roundtable Session 11
  In Roundtable Session: 33.078-16 - The Learning at Hand: Gesture Production in Virtual Pedagogical Agents

Abstract

Gesture benefits comprehension (Hostetter, 2013) and classroom learning (Alibali et al., 2013). As scholars and developers collaborate to design and implement virtual pedagogical agents that provide embodied, multimodal interactions with learners, there is a need to identify principles for effective instructional gestures (Cassell, 2000). The literature covering naturalistic observational studies of teachers, laboratory research, and classroom teaching experiments supports several principles for effective gesture use. Here we focus on three that guide the design of our virtual agent for mathematics instruction (Figure 1): Teaching is multimodal, with gestures playing a central role; pointing and highlighting index ideas to the world; and coordinated sequences of gestures are used to form links, which help connect ideas and forge common ground with students. Together these principles provide guidelines for directing effective pedagogical avatars.

First, teachers regularly use gesture as part of a well-coordinated, multimodal communication to students – along with speech, writing, and object manipulation (see Figure 2) – to address the needs of their students (Edwards, 2009; Flevares and Perry, 2001; Roth, 2001). For example, when teachers communicate connections between mathematical ideas, they almost always express them multimodally (M = 90%, N = 162 across 18 lessons), usually including gestures (Alibali et al., 2014). Furthermore, teachers increase their use of gestures when communicating new as compared to review material (Alibali et al., 2014), and when responding to students’ expression of confusion or trouble spots (Alibali et al., 2013; Nathan & Alibali, 2011).

Second, teachers index ideas and representations to the world through the flexible use of pointing and highlighting acts (Figure 3). These indexical forms relate ideas to perceivable entities and places in the shared learning environment. Teachers frequently use pointing gestures to draw comparisons, and in international comparisons, teachers of higher achieving students do so most frequently (Richland, 2008). Indexing also often goes beyond mere indexing by grounding abstract or unfamiliar ideas and inscriptions in order to scaffold students’ sense-making processes (Nathan, 2008; Walkington et al., 2013). Consistent with scaffolding, the frequency of indexical gestures decreases (fading) as the ideas and inscriptions become more familiar to students (Alibali & Nathan, 2007).

Finally, teachers regularly coordinate multiple gesture-speech acts to establish links among mathematical ideas and representations (Figure 4). Links specify a relationship between mathematical ideas and serve a central role in mathematics lessons, because the meanings of mathematical entities are often developed by establishing relational semantics (Kaput, 1989). In a corpus analysis (Alibali et al., 2014), middle school mathematics teachers produced an average of 10.8 links per hour of instruction, though there was substantial variability (N = 144; SD = 4.81; range 5.9–18.3). Gestures served a linking function most often when mathematical ideas were first mentioned, even if they were part of a review, and especially when students encountered new, highly abstract information, such as a new formalism. Teachers use gesture-rich communication to foster common ground with students by avoiding anticipated breakdowns in comprehension and to institute conversational repair when breakdowns occur (Nathan & Alibali, 2011).

Authors

• Mitchell J. Nathan, University of Wisconsin - Madison

• Martha W. Alibali, University of Wisconsin - Madison