Measuring the Quality of Collaborative Problem-Solving Through Multilayered Audio Analysis

• In Event: Critical Aspects of Computer-Supported Collaborative Learning

Abstract

Objectives or purposes:
The objective of this study is to investigate new ways of assessing verbal social interaction, situated in the center of the PIP model, of adults during collaborative problem-solving (CPS) processes.
 
Perspective(s) or theoretical framework
Research on how the quality of CPS can be most effectively measured is limited (Graesser et al., 2020). One upcoming avenue of measurement is the use of audio data to investigate natural language communication (Lin et al., 2021; Praharaj et al., 2022). This study aims to foster research on multimodal learning analytics by questioning how (a) manual coding with a coding scheme and (b) semi-automatic analysis based on linguistics inquiry and word count (LIWC) and (c) acoustic data analysis can be informative together for the quality of CPS.

Methods, techniques, or modes of inquiry & data sources, evidence, objects, or materials
In this paper we focus on the auditive data captured during a training session on CPS with professionals of several companies in a technology-enhanced learning space. We investigate data collected during the second phase of this training, a ‘selection test for space exploration’ activity performed in groups of 4. The participating groups (n = 15) had to solve as many problems as possible in a limited amount of time.
Conversations were recorded during the group activity, which were then transcribed. Subsequently, three audio analysis methods were used. First, content analysis was done with a framework designed based on literature of Graesser and colleagues (2018) and Sun and colleagues (2022). Second, Linguistic Inquiry and Word Count (LIWC, Pennebaker et al., 2015) was used to assess affective, social and cognitive dimensions of participants’ discourse, as indicators of group processes (e.g., Tausczik & Pennebaker, 2010). Language style matching (LSM) was used to investigate participants’ individual and group behavior (Heuer et al., 2020; Ireland & Pennebaker, 2010). Third, acoustic features such as changes in pitch were used to investigate further group dynamics throughout the session.

Results and/or substantiated conclusions or warrants for arguments/point of view
In what follows we show a selection of results of one of our participating groups. Table 1 gives an overview of the different dimensions for content analysis and LIWC, overall and for each of the problems solved by the team, during the 30-minute activity. These results, for example, show that this group spent only a limited amount of time (14% overall) in maintaining team function and organization. Regarding LIWC, we see similar results: the group used a larger number of words describing cognitive processes, than words indicating affective processes.
Further results will be added in the full paper, including detailed results on the individual level within each group.

Scientific or scholarly significance of the study or work
In this study we investigate the combination of different audio and linguistic analysis methods: 1) content analysis based on CPS coding scheme, 2) linguistic inquiry and word count, and 3) acoustic data analysis of conversations. These techniques give insights in the nature of natural language communication during collaborative problem solving, both on an individual and a group level. Further research is needed to investigate the most useful CPS-quality indicators, which will be informative for the design of dashboards for group awareness.

Authors

• Annelies Raes, KU Leuven
  Presenting Author

• Siem Buseyne, KU Leuven
  Presenting Author

• Kamakshi Rajagopal, KU Leuven
  Presenting Author