Multimodal Timescape: A Tool to Analyze and Visually Represent Students' Multimodal Composition Processes

• In Event: 70.032 - Possibilities and Challenges in New Methods and Frameworks for Multimodal Research

Abstract

Overview and Purpose: New methodologies are required to advance the theory and practice of multimodal composition. We have developed a multimodal matrix approach to provide an integrated view of students’ multimodal composing processes, products, and design perspectives (Authors). We present a matrix analytic framework, highlighting tool and modal affordances and constraints (Kress, 2010). We focus on a multimodal timescape tool (Author2, xx; Authors, xx) to analyze and represent screen capture video data of students’ collaborative composing processes, with the goal of understanding how composers shift modes and processes over time, within and across sessions. We draw on two studies (Author2; Authors), sharing our coding scheme, reliability procedures, examples connecting timescape data with other qualitative data and discuss strengths and limitations.

Theoretical Perspectives: We employ multimodal methods to study multimodal composition. At a broad level, we situate our work within a multiliteracies perspective that posits that all meaning-making is multimodal, and that modes carry affordances and constraints, as situated within specific sociocultural contexts of practice (Kress, 2003; New London Group, 1996). Studies have contributed richly detailed portraits of youth highly engaged in a recursive multimodal process of composition (Gilje, 2011;Hull & Nelson, 2005;Ranker, 2008), where modes are layered into laminated wholes (Leander & Frank, 2006). To understand the complexity of multimodal composition requires use of multimodal methods (Jewitt, 2009) that support cross-case comparison and quantitative process analysis to complement qualitative analysis.

Methods: We draw on 2 studies to demonstrate a multimodal matrix approach and timescape tool. Study 1 included 4 pairs of urban middle school youth composing a digital video folktale (Authors); Study 2 included 3 pairs of urban 12th graders composing a website, hypertext, and podcast in response to literature (Author2). We created a timescape for each pair’s process over several composing sessions. Screen capture videos were coded in Excel for on-screen activity (e.g., searching for images online, editing text, importing music, etc.), predominant mode (e.g., image editing vs. sound editing), and time duration per process. Color-coded bar graphs represented the sequence and proportion of time for each mode/process, displaying processes and patterns.

Findings: A multimodal timescape tool offers new possibilities for reducing and analyzing video process data so that patterns of compositional processes are revealed in relation to mode and across time. Color coding allows the viewer to more easily see modal progressions, and to make comparisons across pairs and multimodal compositions. Study 1 involved two coders for each timescape, yielding inter-rater reliability for coding and time-stamping of 88.8% agreement. In addition to supporting previous research demonstrating the recursive, iterative nature of multimodal composing, this timescape analysis revealed how processes unfolded at a micro-level and were mediated by composing tool(s) and genre requirements. It also showed pairs’ varied modal preferences and compositional processes.

Significance: This work contributes to the development of multimodal methods to support data capture, analysis, and representation of multimodal literacies, and specifically, multimodal composition. If education is to be inclusive of all learners, our methods must also be more inclusive.

Authors

• Bridget Dalton, University of Colorado - Boulder

• Blaine Elizabeth Smith, University of Miami