Recovering Performance in Speech Emotion Recognition from Discrete Tokens via Multi-Layer Fusion and Paralinguistic Feature Integration
Abstract
Discrete speech tokens offer significant advantages for storage and language model integration, but their application in speech emotion recognition (SER) is limited by paralinguistic information loss during quantization. This paper presents a comprehensive investigation of discrete tokens for SER. Using a fine-tuned WavLM-Large model, we systematically quantify performance degradation across different layer configurations and k-means quantization granularities. To recover the information loss, we propose two key strategies: (1) attention-based multi-layer fusion to recapture complementary information from different layers, and (2) integration of openSMILE features to explicitly reintroduce paralinguistic cues. We also compare mainstream neural codec tokenizers (SpeechTokenizer, DAC, EnCodec) and analyze their behaviors when fused with acoustic features. Our findings demonstrate that through multi-layer fusion and acoustic feature integration, discrete tokens can close the performance gap with continuous representations in SER tasks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper systematically investigates the use of discrete tokens for speech emotion recognition, proposing innovative methods to recover lost information through multi-layer fusion and paralinguistic feature integration. The work contributes to advancing SER methodologies, with potential applications in various real-world scenarios.
Methodology
The paper presents a well-structured methodology that systematically investigates the use of discrete tokens for speech emotion recognition (SER). It employs a fine-tuned WavLM-Large model and explores various layer configurations and k-means quantization granularities. The introduction of attention-based multi-layer fusion and the integration of openSMILE features to recover paralinguistic cues are innovative strategies that enhance the approach. The methodology is robust, with clear steps outlined for feature extraction, quantization, and fusion, although the reliance on k-means clustering could be further justified in terms of its effectiveness compared to other clustering techniques.
Results & evaluation
The experiments are comprehensive, utilizing a large-scale dataset (MSP-Podcast) and a well-defined evaluation metric (Macro F1 Score) to assess performance across different configurations. The paper effectively compares the performance of discrete WavLM tokens against various neural audio codecs, providing valuable insights into the effectiveness of the proposed methods. However, the results could benefit from more extensive statistical analysis to validate the significance of the findings.
Reproducibility
The paper lacks detailed implementation specifics, such as hyperparameter settings, code availability, and data preprocessing steps, which are crucial for reproducibility. While the methodology is described, the absence of a public repository or demo limits the ability of other researchers to replicate the results.
Limitations
One limitation is the potential overfitting to the MSP-Podcast dataset, which may not generalize well to other SER tasks or datasets. Additionally, the reliance on discrete tokens may still lead to some information loss that is not fully addressed by the proposed methods. The performance of neural audio codecs is notably lower, which raises questions about their applicability in SER tasks.
Broader impact
The findings of this research have significant implications for the development of efficient SER systems, particularly in applications requiring low-latency processing and storage efficiency, such as mobile devices and real-time communication systems. The integration of paralinguistic features could also enhance emotional intelligence in AI systems, making them more responsive and human-like in interactions. This paper systematically investigates the use of discrete tokens for speech emotion recognition, proposing innovative methods to recover lost information through multi-layer fusion and paralinguistic feature integration. The work contributes to advancing SER methodologies, with potential applications in various real-world scenarios.
Author of this paper? Send a correction.