Do Gestural Interfaces Promote Thinking? Embodied Interaction: Congruent Gestures and Direct Touch Promote Performance in Math

• In Event: Roundtable Session 123
  In Roundtable Session: Grounded Embodied Cognition in the K-12 Classroom: Innovations and Explorations in Practice

Abstract

Overview
Embodied interaction involving digital devices is based on the theory of grounded cognition. Embodied interaction with gestural interfaces involves more of our senses than traditional (mouse-based) interfaces, and in particular includes direct touch and physical movement, which are believed to help retain the knowledge that is being acquired. There is growing evidence that spontaneous gestures affect thought and possibly learning. The author was interested to explore whether designed gestures (for gestural interfaces) affect thought. It was hypothesized that the use of congruent gestures helps construct better mental representations and mental operations to solve problems (Gestural Conceptual Mapping). There is also evidence that physical manipulation of objects can benefit cognition and learning; it was therefore also hypothesized that manipulating objects through direct touch on the screen supports performance.

Can action support cognition? Can direct touch support performance?
These hypotheses were addressed by observing children’s performance in arithmetic and numerical estimation. Arithmetic is a discrete task, and should be supported by discrete rather than continuous actions. Estimation is a continuous task, and should be supported by continuous rather than discrete actions. Children used either a gestural interface (multi-touch, e. g., iPad) or a traditional keyboard/mouse interface. The actions either mapped congruently to the cognition (continuous action for estimation and discrete action for arithmetic), or not.

Participants
The researcher recruited 107 subjects (60 boys) who were six and seven years old from 1st and 2nd grade. Children were recruited from public schools in a low-SES area of NYC.

Materials
Two educational applications were developed to allow interaction and learning with two math concepts. The learned concepts were of discrete-change problems, such as counting blocks, vs. concepts of continuous-change problems, such as number-line estimation. The gestural interface was a 10” multi-touch iPad device and the traditional interface was a Macbook Pro laptop, which requires the use of a mouse.

Variables and design
It was a 2x2 between subjects design. There were four conditions and two variables: Gestural Conceptual Mapping, the mapping the gesture to the learned concept, and Direct-Touch Input.

Results and Conclusions
1. Action supports cognition if the action is congruent with the thinking
Gestural Conceptual Mapping (congruent gestures) promotes performance. Children who used discrete gestures to solve arithmetic problems, and continuous gestures to solve number estimation, performed better.

2. Touch interfaces are more efficient than mouse interfaces and promote using advanced strategies
Children who used a touch interface applied an advanced “count on” strategy for arithmetic, more frequently than, children using a mouse. Children who used a touch interface spent less time on tasks.

Significance
More empirical studies by cognitive psychologists, educators, and human-computer interaction researchers are needed to answer the question of whether the design of gestural interfaces could have effects on learning. These gestural interfaces will soon become an integral component among educational tools for children. Gestural interfaces could provide more concrete experiences because they require us to use our body in a more direct way, thereby possibly creating more meaningful learning experiences for young children.

Author

• Ayelet Segal, Teachers College, Columbia University