Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition
Abstract
Speech Emotion Recognition (SER) in real-world scenarios remains challenging due to severe class imbalance and the prevalence of spontaneous, natural speech. While recent approaches leverage self-supervised learning (SSL) representations and multimodal fusion of speech and text, most existing methods apply supervision only at the final classification layer, limiting the discriminative power of intermediate representations. In this work, we propose Crab (Contrastive Representation and Multimodal Aligned Bottleneck), a bimodal Cross-Modal Transformer architecture that integrates speech representations from WavLM and textual representations from RoBERTa, together with a novel \textit{Multi Layer Contrastive Supervision} (MLCS) strategy. MLCS injects multi-positive contrastive learning signals at multiple layers of the network, encouraging emotionally discriminative representations throughout the model without introducing additional parameters at inference time. To further address data imbalance, we adopt weighted cross-entropy during training. We evaluate the proposed approach on three benchmark datasets covering different degrees of emotional naturalness: IEMOCAP, MELD, and MSP-Podcast 2.0. Experimental results demonstrate that Crab consistently outperforms strong unimodal and multimodal baselines across all datasets, with particularly large gains under naturalistic and highly imbalanced conditions. These findings highlight the effectiveness of \textit{Multi Layer Contrastive Supervision} as a general and robust strategy for SER. Official implementation can be found in https://github.com/AI-Unicamp/Crab.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents Crab, a multimodal SER framework that effectively integrates speech and text representations through a novel contrastive learning strategy, achieving significant performance improvements in emotion recognition tasks. The innovative approach and rigorous evaluation contribute meaningfully to the field of speech emotion recognition, addressing key challenges in real-world applications.
Methodology
The proposed methodology introduces a novel Cross-Modal Transformer architecture, integrating speech and text representations while employing Multi Layer Contrastive Supervision (MLCS) to enhance emotion recognition. This approach is innovative as it applies contrastive learning at multiple layers, which is not common in existing SER frameworks. The use of weighted cross-entropy to address class imbalance further strengthens the methodology, making it robust for real-world applications.
Results & evaluation
The experimental evaluation is comprehensive, utilizing three benchmark datasets (IEMOCAP, MELD, and MSP-Podcast 2.0) that vary in emotional naturalness. The results consistently demonstrate superior performance of the Crab model compared to strong baselines, particularly in challenging naturalistic scenarios with class imbalance. The use of multiple evaluation metrics (UAR and WAR) provides a well-rounded assessment of model performance.
Reproducibility
The paper includes a link to the official implementation on GitHub, which is crucial for reproducibility. However, specific implementation details such as hyperparameters and training configurations could be more explicitly stated to facilitate easier replication by other researchers.
Limitations
One limitation is the reliance on specific datasets, which may not fully capture the diversity of emotional expressions in real-world scenarios. Additionally, while the model shows robustness to class imbalance, the performance on unseen speakers in naturalistic conditions could be further explored.
Broader impact
The findings have significant implications for applications in human-computer interaction, customer service, and online education, where understanding emotional cues can enhance user experience. The proposed model's ability to handle class imbalance makes it particularly valuable for deploying SER systems in real-world contexts. The paper presents Crab, a multimodal SER framework that effectively integrates speech and text representations through a novel contrastive learning strategy, achieving significant performance improvements in emotion recognition tasks. The innovative approach and rigorous evaluation contribute meaningfully to the field of speech emotion recognition, addressing key challenges in real-world applications.
Author of this paper? Send a correction.