Explainable Transformer-CNN Fusion for Noise-Robust Speech Emotion Recognition
Abstract
Speech Emotion Recognition (SER) systems often degrade in performance when exposed to the unpredictable acoustic interference found in real-world environments. Additionally, the opacity of deep learning models hinders their adoption in trust-sensitive applications. To bridge this gap, we propose a Hybrid Transformer-CNN framework that unifies the contextual modeling of Wav2Vec 2.0 with the spectral stability of 1D-Convolutional Neural Networks. Our dual-stream architecture processes raw waveforms to capture long-range temporal dependencies while simultaneously extracting noise-resistant spectral features (MFCC, ZCR, RMSE) via a custom Attentive Temporal Pooling mechanism. We conducted extensive validation across four diverse benchmark datasets: RAVDESS, TESS, SAVEE, and CREMA-D. To rigorously test robustness, we subjected the model to non-stationary acoustic interference using real-world noise profiles from the SAS-KIIT dataset. The proposed framework demonstrates superior generalization and state-of-the-art accuracy across all datasets, significantly outperforming single-branch baselines under realistic environmental interference. Furthermore, we address the ``black-box" problem by integrating SHAP and Score-CAM into the evaluation pipeline. These tools provide granular visual explanations, revealing how the model strategically shifts attention between temporal and spectral cues to maintain reliability in the presence of complex environmental noise.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of a Hybrid Transformer-CNN framework for noise-robust speech emotion recognition that effectively combines temporal and spectral features while providing explainability through SHAP and Score-CAM. This research significantly advances the state-of-the-art in SER by addressing both performance and interpretability, making it highly relevant for practical applications in diverse acoustic environments.
Methodology
The proposed methodology combines a Transformer model (Wav2Vec 2.0) with a CNN to create a dual-stream architecture that captures both long-range temporal dependencies and noise-resistant spectral features. The integration of an Attentive Temporal Pooling mechanism is a significant innovation, allowing the model to focus on relevant features dynamically. The methodology is well-structured, with a clear explanation of the noise injection protocols and feature extraction processes, which are crucial for achieving robustness in real-world scenarios.
Results & evaluation
The experiments are comprehensive, utilizing multiple benchmark datasets (RAVDESS, TESS, SAVEE, CREMA-D) and a custom noise dataset (SAS-KIIT) to rigorously test the model's performance under various conditions. The results demonstrate superior accuracy and robustness compared to single-branch baselines, particularly in non-stationary noise environments. The ablation study further validates the contributions of each component of the architecture, providing strong evidence for the effectiveness of the proposed approach.
Reproducibility
While the paper provides a detailed description of the methodology and experiments, it lacks specific implementation details such as code availability or links to datasets, which are critical for reproducibility. The absence of a demo URL or project repository limits the ability for other researchers to replicate the findings or build upon this work.
Limitations
One limitation is the reliance on specific datasets, which may not fully represent all real-world acoustic environments. Additionally, while the model shows robustness against various noise types, further testing against a broader range of environmental conditions could provide a more comprehensive evaluation of its generalizability. The paper also does not discuss the computational efficiency of the proposed model, which could be a concern for deployment in real-time applications.
Broader impact
The research has significant implications for applications in human-computer interaction, mental health monitoring, and customer service, where understanding emotional intent from speech is critical. By addressing the robustness and interpretability of SER systems, this work paves the way for more reliable and trustworthy AI applications in sensitive domains. The main contribution of this paper is the development of a Hybrid Transformer-CNN framework for noise-robust speech emotion recognition that effectively combines temporal and spectral features while providing explainability through SHAP and Score-CAM. This research significantly advances the state-of-the-art in SER by addressing both performance and interpretability, making it highly relevant for practical applications in diverse acoustic environments.
Author of this paper? Send a correction.