Distilled HuBERT for Mobile Speech Emotion Recognition: A Cross-Corpus Validation Study
Abstract
Speech Emotion Recognition (SER) has significant potential for mobile applications, yet deployment remains constrained by the computational demands of state-of-the-art transformer architectures. This paper presents a mobile-efficient SER system based on DistilHuBERT, a distilled and 8-bit quantized transformer that achieves approximately 92% parameter reduction compared to full-scale Wav2Vec 2.0 models while maintaining competitive accuracy. We conduct a rigorous 5-fold Leave-One-Session-Out (LOSO) cross-validation on the IEMOCAP dataset to ensure speaker independence, augmented with cross-corpus training on CREMA-D to enhance generalization. Cross-corpus training with CREMA-D yields a 1.2% improvement in Weighted Accuracy, a 1.4% gain in Macro F1-score, and a 32% reduction in cross-fold variance, with the Neutral class showing the most substantial benefit at 5.4% F1-score improvement. Our approach achieves an Unweighted Accuracy of 61.4% with a quantized model footprint of only 23 MB, representing approximately 91% of the Unweighted Accuracy of a full-scale baseline. Cross-corpus evaluation on RAVDESS reveals that the theatrical nature of acted emotions causes predictions to cluster by arousal level rather than by specific emotion categories - happiness predictions systematically bleed into anger predictions, and sadness predictions bleed into neutral predictions, due to acoustic saturation when actors prioritize clarity over subtlety. Despite this theatricality effect reducing overall RAVDESS accuracy to 46.64%, the model maintains robust arousal detection with 99% recall for anger, 55% recall for neutral, and 27% recall for sadness. These findings demonstrate a Pareto-optimal tradeoff between model size and accuracy, enabling practical affect recognition on resource-constrained mobile devices.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper demonstrates a significant advancement in mobile Speech Emotion Recognition by leveraging a distilled transformer architecture, achieving a balance between model efficiency and accuracy. The methodology is rigorous, and the findings contribute valuable insights into the challenges of cross-corpus generalization in emotion recognition tasks.
Methodology
The paper presents a well-structured methodology that emphasizes mobile deployment and cross-corpus robustness. The use of DistilHuBERT for model compression is innovative, and the 5-fold Leave-One-Session-Out (LOSO) cross-validation approach is commendable for ensuring speaker independence. The integration of CREMA-D for cross-corpus training is a strategic move to enhance generalization, although the authors acknowledge the challenges posed by the differing nature of the datasets. The acoustic processing and data augmentation techniques are thorough, addressing potential biases and enhancing model robustness.
Results & evaluation
The experimental framework is robust, with clear metrics for evaluation. The results indicate a significant improvement in performance metrics with cross-corpus training, and the detailed analysis of class-wise performance provides valuable insights into the model's strengths and weaknesses. However, the overall accuracy on the RAVDESS dataset highlights the challenges of generalization across different emotional expression styles.
Reproducibility
The paper mentions that all experimental code and trained models are publicly available, which is crucial for reproducibility. However, specific URLs for accessing these resources are not provided, which could hinder the ease of replication by other researchers.
Limitations
The paper acknowledges limitations such as the reduced accuracy on the RAVDESS dataset due to the theatrical nature of the emotions expressed, which may not align with the model's training data. Additionally, while the proposed deployment architecture is promising, it requires empirical validation to assess its effectiveness in real-world applications.
Broader impact
The work has significant implications for mobile applications in mental health monitoring and adaptive human-computer interaction. By addressing the computational demands of SER systems, this research paves the way for more accessible and privacy-preserving emotion recognition technologies. This paper demonstrates a significant advancement in mobile Speech Emotion Recognition by leveraging a distilled transformer architecture, achieving a balance between model efficiency and accuracy. The methodology is rigorous, and the findings contribute valuable insights into the challenges of cross-corpus generalization in emotion recognition tasks.
Other indexed descriptions from the archive
Mobile-Efficient Speech Emotion Recognition Using DistilHuBERT: A Cross-Corpus Validation Study
Abstract
Speech Emotion Recognition (SER) has significant potential for mobile applications, yet deployment remains constrained by the computational demands of state-of-the-art transformer architectures. This paper presents a mobile-efficient SER system based on DistilHuBERT, a distilled and 8-bit quantized transformer that achieves 92% parameter reduction compared to full-scale Wav2Vec 2.0 models while maintaining competitive accuracy. We conduct a rigorous 5-fold Leave-One-Session-Out (LOSO) cross-validation on the IEMOCAP dataset to ensure speaker independence, augmented with cross-corpus training on CREMA-D to enhance generalization. Cross-corpus training with CREMA-D yields a 1.2% improvement in Weighted Accuracy, a 1.4% gain in Macro F1-score, and a 32% reduction in cross-fold variance, with the Neutral class showing the most substantial benefit at 5.4% F1-score improvement. Our approach achieves an Unweighted Accuracy of 61.4% with a quantized model footprint of only 23 MB, representing approximately 91% of full-scale baseline performance. Cross-corpus evaluation on RAVDESS reveals that the theatrical nature of acted emotions causes predictions to cluster by arousal level rather than valence: happiness is systematically confused with anger due to acoustic saturation in high-energy expressions. Despite this theatricality effect reducing overall RAVDESS accuracy to 43.29%, the model maintains robust arousal detection with 97% recall for anger and 64% for sadness. These findings establish a Pareto-optimal tradeoff between model size and accuracy, enabling practical affect recognition on resource-constrained mobile devices.
AI research summary
This paper establishes a mobile-efficient SER system using DistilHuBERT, demonstrating a significant contribution to the field by addressing both computational efficiency and generalization challenges in emotion recognition. The rigorous methodology and experimental validation enhance its relevance and potential impact in real-world applications.
AI methodology analysis
The methodology presented in this paper is robust, emphasizing a mobile-efficient architecture through the use of DistilHuBERT, which is a significant advancement in the field of Speech Emotion Recognition (SER). The authors adopt a strict 5-fold Leave-One-Session-Out (LOSO) cross-validation protocol to mitigate speaker leakage, enhancing the reliability of their results. The integration of the CREMA-D dataset for cross-corpus training is a novel approach that addresses generalization issues, although it introduces complexities due to differences in emotional expressiveness. The use of Adaptive Focal Loss to handle class imbalance is a thoughtful addition that demonstrates an understanding of the challenges in SER.
AI results analysis
The experimental framework is well-structured, utilizing the IEMOCAP dataset as the primary benchmark while ensuring speaker independence through rigorous validation protocols. The results indicate a 1.2% improvement in Weighted Accuracy and a significant reduction in cross-fold variance, showcasing the effectiveness of cross-corpus training. However, the performance drop on the RAVDESS dataset highlights the challenges of domain adaptation, particularly in distinguishing emotions in acted versus naturalistic speech. The findings are presented clearly, with appropriate metrics and comparisons to existing models.
AI reproducibility analysis
The authors have made efforts to ensure reproducibility by providing access to their experimental code and trained models, although specific URLs are not mentioned in the paper. This commitment to transparency is commendable and facilitates further research in the field.
AI limitations analysis
One notable limitation is the performance drop observed on the RAVDESS dataset, attributed to the theatricality of the emotions expressed, which may not generalize well to real-world applications. Additionally, while the paper discusses the potential for future work, it does not explore the implications of the identified theatricality gap in depth. The reliance on a distilled transformer architecture may also limit the model's ability to capture subtle emotional nuances.
AI broader impact analysis
The implications of this research are significant, as it paves the way for practical applications of SER in mobile devices, enhancing user interaction in various domains such as mental health monitoring and customer service. The focus on privacy-preserving, low-latency emotion recognition aligns well with current trends in AI ethics and user-centered design. This paper establishes a mobile-efficient SER system using DistilHuBERT, demonstrating a significant contribution to the field by addressing both computational efficiency and generalization challenges in emotion recognition. The rigorous methodology and experimental validation enhance its relevance and potential impact in real-world applications.
Author of this paper? Send a correction.