Transmediation in Children's Virtual Reality Gaming

• In Event: 43.053 - Materiality Meets Multimodality: The Forgotten Role of Material Objects in Digital Literacies

Abstract

Objective
Existing research has examined how young children shift meanings across sign systems long before they have mastered formal writing skills, yet virtual reality environments offer extraordinary new affordances for users to translate semiotic material within and across virtual and material spaces that have not been researched.

Theoretical framework
In a digital age, children and youth are socialised into a broadened and sophisticated repertoires of digital media conventions, which potentially include new virtual reality gaming environments. The study extends knowledge of transmediation first coined by Suhor (1992), and extended by Semali (2002), Siegel (2006) and others (e.g. Author 3, 2011). Every sign system or mode has unique organisational principles, involving elements or conventions that do not have precisely equivalent meanings when translated across modes. The term “modes” and “sign-systems” are used interchangeably as socially and culturally shaped resources or semiotic structures for making meaning. They are organised, regular, socially specific means of representation, such as writing, drawing, dance, image, music or mathematics (Author 3, 2011). The lack of equivalence between modes is a catalyst for transmediation. This presentation extends theories of transmediation that have predominantly explored speech, drawing and writing, to advance new understandings about children’s engagement in transmediation in the context of virtual reality gaming.

Methods and Data Sources
Nine to twelve year-old children were given the task of creating three dimensional, virtual texts with Google Tilt Brush using a virtual reality headset and motion sensors. The creation had to represent an idea or element from their own written story, such as illustrating part of the story setting or a character. A multimodal analysis of the video recording was conducted, focusing on how the children shifted meanings from their written stories to 3D virtual reality scenes. The researcher also analysed the auto-recorded, minute-by-minute screencast of the children's’ designing in Google Tilt Brush. Insights on the children’s process of transmediation was also gathered through the analysis of a think-aloud protocol conducted with the children about their virtual creations as they “painted” using the VR headset and motion sensors.

Results
The research demonstrates that translating semiotic content from the materiality of written stories to 3D virtual reality design highlighted a lack of equivalence between the affordances of the sign systems and some similarities that both enabled and constrained the children’s communication of meaning.

Significance
The increasing availability of new digital technologies for creative representation of ideas, such as virtual reality, generates new possibilities for transmediation in young children’s multimodal compositions that are yet to be explored from this semiotic perspective. Providing opportunities for students to translate literary content to virtual reality environments can enable them to go beyond the simple reproduction of story content to transform meaning and knowledge through a different and currently unexplored virtual materiality.

Author

• Kathy Ann Mills, Australian Catholic University