Potential and Pitfalls of Audio-as-Data, from Alignment to Multimodal Models

• In Event: Preprocessing, Annotation, and Interpretation in Computational Social Science

Abstract

This paper discusses the challenges and potential of using multimodal data in political settings. Researchers in political science have been relatively slow to adopt multimodal data analysis as it requires an extended set of skills in data science and machine learning, particularly in text, audio and video processing, as well as high computational power to perform analyses. Nevertheless, political science is a field rich in a great variety of multimodal sources of information, from televised debates to parliamentary briefings or TV interviews. Advances in text-as-data approaches and the availability of Application User Interfaces (APIs) to obtain data from various sources have made it possible to analyze massive amounts of data in ways that were not feasible before. This paper argues that the audio and video sources in political speeches can also provide valuable insights into the speaker's emotional state and should thus be analyzed alongside text. The paper focuses on audio analysis and aims to address the challenges encountered when trying to analyse audio sources and to explore the potential of multimodal data analysis in the field of political science.
One of the central elements of this paper is the use of forced alignment techniques to match text and audio/video sources so they can be analyzed concurrently. Whereas nowadays many video data sources come with transcripts that include timestamps, this is not the case for older sources, like TV interviews or debates, where both audio, video and text sources exist, but they have not been aligned. Even when timestamps are provided, such as in some governmental repositories with parliamentary briefings or sessions, these timestamps are only meant to indicate the beginning of the session, and not when each speaker talks. Automatic transcription tools have also been proposed to circumvent this issue as they generally provide sentence-level timestamps. However, they can be the source of several systematic errors, especially when recording names that are not common in the anglosphere. If the researcher already has access to high quality human-transcribed text, aligning text and audio might be a more suitable choice. In this paper, therefore, we review the technique of forced alignment using a dynamic time warping (DTW) algorithm to match text and audio with high accuracy to enable multimodal analysis.
In addition to this, this paper also reviews how audio features can be used in multimodal machine learning algorithms to extract valuable insights from political speeches. For instance, one of the most commonly used features in audio analysis is Mel-frequency cepstral coefficients (MFCCs), which are used to represent the spectral envelope of audio signals and are used in both forced alignment and audio classification models, particularly in speech recognition, speaker identification and emotion recognition. We provide explanations to help users understand MFCCs for both experienced researchers and those who want to start working with audio. We give an overview of how MFCCs are calculated, the underlying assumptions, and we also explain how they can be used as input features in machine learning models, such as convolutional neural networks (CNNs) and bidirectional long short-term memory (BLSTM) arrays.
Overall, this paper discusses the challenges and potentials of using multimodal data in political science. We focus on the technique of forced alignment and the use of audio features in machine learning algorithms, such as MFCCs, to extract valuable insights from political speeches, and provide explanations for understanding and implementing these techniques. To summarise, this paper is aimed at bridging the gap between computer science and political science in the field of multimodal data analysis to help researchers who want to start working with audio data and video data in political science.

Authors

• Rafael Mestre, University of Southampton

• Matt Ryan, University of Southampton