Identifying Pivotal Moments in Individual and Group Innovation Processes: Considerations of a Temporal Analytic Approach

• In Event: Connecting the Moving Dots: Comparing Approaches to Coordinating Temporal Analyses of Groups and Individuals

Abstract

Objectives
We illustrate a method of analyzing individuals’ actions in the context of and as contributions to collaborative problem solving, examining the design and instrumentation of an exploratory study of collaborative innovation. We describe how methods employed address issues arising from the use of temporal analyses including the validity of inferences made using these methods and the identification of informative patterns of behavior, especially when analyses integrate multiple data types.

The study relied on: 1) a framework of innovation based on behaviors identified as being conducive to collaborative innovation via literature review (Vahey, Brecht, Patton, Rafanan, and Cheng, in press); 2) the assumption that temporal analyses of multiple data types would be necessary in order to identify how individual actions relate to one another and to the development of collective knowledge and problem solving strategies (Arrow, McGrath, & Berdahl, 2000; Stahl, 2005; Azevedo, 2008; Reimann, 2009); and 3) an Evidence-Centered Design (ECD; Mislevy, Steinberg, & Almond, 2003), process to align the design of the task presented to participants with the coding and analysis of participant and group behaviors. This approach ensured the coherence of our methods, grounding analytic choices and data interpretation.

Data Sources
Developed in the Second Life virtual environment, the task required participants to understand the connectivity and cost of a variety of transceivers to create a communication network serving an island resort. Participants were equipped with unique information needed to solve the task. Three adults were introduced to the environment and the task by a researcher during each data collection session. Participants’ actions (clicks) and online chat were collected and analyzed.

In many instances phenomena of interest (e.g., knowledge construction) cannot be directly observed in sequences of user-system interactions, because they comprise patterns of interactions, higher-level “hidden states” as probabilistic distributions over the observed values. In this study, Hidden Markov Modeling (HMM) was used to posit ‘states’, corresponding to identified innovation behaviors based on a constructed data set that coded units of both click and chat data simultaneously.

Results
Analyses revealed crucial moments in the sessions when behaviors changed significantly and, in the case of Group 3, ultimately lead to a problem solution. Patterns could not be detected using count-and-code analytic methods that report behavior frequencies nor with analyses of either click or chat data individually. A pivotal moment was also identified using other analyses including Latent Semantic Analysis; these will be reported in this presentation as will individual participant contributions to this phenomenon.

Significance
The development and evaluation of methodological approaches to analyzing complex group phenomena is a pervasive challenge in our field. The principled, coordinated use of: 1) ECD to improve the validity of inferences made using temporal analytic methods, 2) of HMM to identify significant patterns of behavior, and 3) of an integrated coding and analysis of click and chat data represent one such approach that can add to the research community’s tools and understanding of their use.

Authors

• Britte Haugan Cheng, SRI International

• Philip J. Vahey, SRI International

• John J. Brecht, SRI International

• Charles Patton, SRI International

• Ken Rafanan, SRI International