MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages
Abstract
We present MERaLiON-SER, a robust speech emotion recognition model de- signed for English and Southeast Asian languages. The model is trained using a hybrid objective combining weighted categorical cross-entropy and Concordance Correlation Coefficient (CCC) losses for joint discrete and dimensional emotion modelling. This dual approach enables the model to capture both the distinct categories of emotion (like happy or angry) and the fine-grained, such as arousal (intensity), valence (positivity/negativity), and dominance (sense of control), lead- ing to a more comprehensive and robust representation of human affect. Extensive evaluations across multilingual Singaporean languages (English, Chinese, Malay, and Tamil ) and other public benchmarks show that MERaLiON-SER consistently surpasses both open-source speech encoders and large Audio-LLMs. These results underscore the importance of specialised speech-only models for accurate paralin- guistic understanding and cross-lingual generalisation. Furthermore, the proposed framework provides a foundation for integrating emotion-aware perception into future agentic audio systems, enabling more empathetic and contextually adaptive multimodal reasoning.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of MERaLiON-SER, a robust speech emotion recognition model that excels in multilingual and culturally diverse settings. The comprehensive evaluation and innovative methodology underscore its potential to advance the field of affective computing, particularly in underrepresented languages and emotional expressions.
Methodology
The methodology presented in MERaLiON-SER is robust and innovative, utilizing a hybrid loss function that combines weighted categorical cross-entropy and Concordance Correlation Coefficient (CCC) losses. This dual approach allows the model to effectively capture both discrete and dimensional aspects of emotion, which is a significant advancement in speech emotion recognition (SER). The architecture employs a Whisper-Medium encoder and integrates attention-based pooling and ECAPA-TDNN modules, enhancing the model's ability to generalize across languages and emotional expressions. The use of Low-Rank Adaptation (LoRA) for efficient fine-tuning is particularly noteworthy, as it balances computational efficiency with performance.
Results & evaluation
The experimental evaluation is extensive, covering multiple datasets, including proprietary and public benchmarks. The model demonstrates superior performance across various languages, achieving the highest Unweighted Average Recall (UAR) compared to both open-source and closed-source models. The rigorous evaluation on multilingual datasets, including the SG-ECMT and public emotion datasets, reinforces the model's robustness and generalization capabilities. The results are well-presented, providing clear comparisons with baseline models, which adds credibility to the findings.
Reproducibility
The paper provides detailed implementation information, including model architecture, training configurations, and data augmentation techniques. However, while the methodology is well-documented, the lack of a publicly available code repository limits reproducibility. The authors mention that the model is available on Hugging Face, which aids reproducibility to some extent, but a comprehensive code release would enhance this aspect significantly.
Limitations
The model's reliance on pseudo-labeled data for certain languages may introduce noise and bias, which could affect performance. Additionally, the model is limited to seven emotion classes and may not generalize well to more complex emotional contexts, such as sarcasm detection. The authors acknowledge these limitations and suggest future work to address them, particularly in enhancing the dataset with more manually labeled examples.
Broader impact
The MERaLiON-SER model has significant implications for applications in affective computing, particularly in developing empathetic conversational agents and improving human-machine interactions. Its ability to recognize emotions across multiple Southeast Asian languages positions it as a valuable tool in diverse cultural contexts, potentially enhancing user experiences in virtual assistants and mental health monitoring systems. The main contribution of this paper is the development of MERaLiON-SER, a robust speech emotion recognition model that excels in multilingual and culturally diverse settings. The comprehensive evaluation and innovative methodology underscore its potential to advance the field of affective computing, particularly in underrepresented languages and emotional expressions.
Author of this paper? Send a correction.