Prosody as Supervision: Bridging the Non-Verbal--Verbal for Multilingual Speech Emotion Recognition
Abstract
In this work, we introduce a paralinguistic supervision paradigm for low-resource multilingual speech emotion recognition (LRM-SER) that leverages non-verbal vocalizations to exploit prosody-centric emotion cues. Unlike conventional SER systems that rely heavily on labeled verbal speech and suffer from poor cross-lingual transfer, our approach reformulates LRM-SER as non-verbal-to-verbal transfer, where supervision from a labeled non-verbal source domain is adapted to unlabeled verbal speech across multiple target languages. To this end, we propose NOVA ARC, a geometry-aware framework that models affective structure in the Poincaré ball, discretizes paralinguistic patterns via a hyperbolic vector-quantized prosody codebook, and captures emotion intensity through a hyperbolic emotion lens. For unsupervised adaptation, NOVA-ARC performs optimal transport based prototype alignment between source emotion prototypes and target utterances, inducing soft supervision for unlabeled speech while being stabilized through consistency regularization. Experiments show that NOVA-ARC delivers the strongest performance under both non-verbal-to-verbal adaptation and the complementary verbal-to-verbal transfer setting, consistently outperforming Euclidean counterparts and strong SSL baselines. To the best of our knowledge, this work is the first to move beyond verbal-speech-centric supervision by introducing a non-verbal-to-verbal transfer paradigm for SER.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This work presents a groundbreaking approach to multilingual speech emotion recognition by introducing a non-verbal-to-verbal transfer paradigm, significantly advancing the field by addressing the challenges of low-resource settings and enhancing the robustness of emotion recognition systems.
Methodology
The methodology introduces a novel approach to multilingual speech emotion recognition (SER) by leveraging non-verbal vocalizations as a source of supervision. The proposed NOVA-ARC framework utilizes hyperbolic geometry to model affective structures and employs optimal transport for aligning emotion prototypes with target utterances. This innovative non-verbal-to-verbal transfer paradigm is a significant departure from traditional SER methods that rely on labeled verbal data, showcasing a well-thought-out architecture that integrates various advanced techniques.
Results & evaluation
The experiments are comprehensive, utilizing multiple datasets that span various languages and emotional expressions. The results demonstrate that NOVA-ARC consistently outperforms traditional Euclidean models and strong self-supervised learning baselines, validating the effectiveness of the proposed approach across different settings. The inclusion of diverse corpora and rigorous evaluation metrics strengthens the findings.
Reproducibility
The paper provides detailed descriptions of the models and training procedures, including hyperparameters and dataset configurations. However, the absence of a publicly available implementation or code repository may hinder full reproducibility. The authors could enhance reproducibility by sharing their code and trained models.
Limitations
The study acknowledges limitations in evaluating spontaneous conversational speech and the potential challenges when emotion categories are closely related. The reliance on publicly available datasets may also restrict the generalizability of the findings to real-world applications.
Broader impact
The proposed framework has significant implications for developing more robust and scalable SER systems, particularly in low-resource settings. By decoupling emotional expression from language-specific cues, this research could enhance the accessibility of emotion recognition technologies across diverse linguistic backgrounds and applications, such as conversational agents and assistive technologies. This work presents a groundbreaking approach to multilingual speech emotion recognition by introducing a non-verbal-to-verbal transfer paradigm, significantly advancing the field by addressing the challenges of low-resource settings and enhancing the robustness of emotion recognition systems.
Author of this paper? Send a correction.