Using Gesture Features to Assess Learning in a Whole-Body Science Simulation

• In Event: Transfer for Learning Outside the Classroom: Informal Contexts to Examine STEM Learning Skills and Strategies

Abstract

There has been significant interest in leveraging learners’ embodied interactions for teaching critical STEM concepts (Author et al., 2016a; Stieff et al., 2016). A particularly challenging concept that cuts across STEM domains is non-linear growth, understanding where it is present and how it differs from linear growth (Tretter et al., 2006). We designed an embodied environment for learning about non-linear growth called ELASTIC3S that allows learners to control different science simulations with user-defined whole-body gestures (e.g., hand waving, kicking). Technological advances in gesture recognition allow for the creation of “mixed reality” environments that can track and respond to students’ gestures, but there has so far been little investigation into how the gestures learners perform in these environments relate to their subsequent learning, and how the gestures themselves can be used for the purposes of assessment.

ELASTIC3S—created with the Unity engine and Microsoft Kinect V2—was designed to facilitate learning transfer by having students develop gestures to express ideas about linear/non-linear growth in one domain (earthquakes and the Richter Scale) and then apply them to another domain (acids/bases and the pH scale). This paper focuses on the behavior of 16 undergraduate students who used the earthquake simulation to complete 5 tasks in a 30 minutes session. Participants completed a pretest and a posttest comprised of similar questions that assessed their understanding of earthquakes and linear/non-linear growth in both the context of earthquakes and new contexts. Participants completed an engagement survey probing their responses to the experience in terms of cognition, learning, enjoyment, attention, and enjoyment.

We were interested in how these more traditional learning measures related to three spatial dimensions of the gestures they performed while engaging with the simulation environment. We examined two kinematic features that were computed using the skeleton’s joint positions as provided by the Kinect sensor: gesture speed (magnitude of the velocity of a given position) and gesture volume (product of the euclidean distance between the maximum and minimum points of given joint positions at each dimension). We then used the mean of each feature as a generalized aggregation of the feature over each task. The results of Pearson correlations showed significant positive associations between the volume feature of the final task and engagement: e.g., Volume and Learning_engagement: r = .621 (p = .023), Volume and Cognitive_engagement: r = .560 (p = .047). The speed feature of the final task showed significant negative associations with Conceptual change (r = -.657, p = .015) and Exponential change (r = -.584, p = .036). As such, gesture volume is only associated with the student’s engagement, while gesture speed is associated with their learning gains (i.e., slower gestures were associated with greater learning). This suggests interesting lines of follow-up inquiry on the causal relationship between gesture features and learning. The analytical approach used in this study indicates the potential of kinematic features as key indicators of the quality of learner perceptions and comprehension, and the potential need for gestural interaction guidance, which can further support their learning and transfer to other domains.

Authors

• Jina Kang, University of Illinois at Urbana-Champaign

• Robb Lindgren, University of Illinois at Urbana-Champaign

• Michael Junokas, University of Illinois at Urbana-Champaign