AcoustEmo: Open-Vocabulary Emotion Reasoning via Utterance-Aware Acoustic Q-Former
Abstract
Multimodal Large Language Models (MLLMs) excel in Open-Vocabulary (OV) emotion recognition but often neglect fine-grained acoustic modeling. Existing methods typically use global audio encoders, failing to capture subtle, local temporal dynamics like micro-prosody and intonation shifts within individual utterances. To address this, we propose AcoustEmo, a time-sensitive MLLM featuring a novel Utterance-Aware Acoustic Q-Former. Our approach utilizes a timestamp-synchronized sliding window to dynamically extract segment-level audio tokens instead of coarse global representations. This enables the model to explicitly trace the temporal evolution of subtle acoustic clues and capture deep contextual dependencies in dialogues. Experiments on the Explainable Multimodal Emotion Recognition (EMER) task show that AcoustEmo significantly enhances complex emotion reasoning, outperforming baselines while maintaining robust contextual accuracy.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents AcoustEmo, a time-sensitive MLLM that significantly enhances open-vocabulary emotion reasoning by capturing local acoustic dynamics through a novel Utterance-Aware Acoustic Q-Former. This work is a meaningful contribution to the field of multimodal emotion recognition, addressing critical gaps in existing methodologies and demonstrating substantial technical advancements.
Methodology
The proposed methodology introduces a novel architecture, AcoustEmo, which leverages an Utterance-Aware Acoustic Q-Former to address the limitations of traditional global audio encoders in capturing fine-grained acoustic details. The use of a timestamp-synchronized sliding window for dynamic extraction of segment-level audio tokens is innovative, allowing the model to maintain semantic coherence between audio and text modalities. This approach is well-justified and effectively targets the nuances of emotion conveyed through micro-prosody and intonation shifts, which are critical for accurate emotion recognition.
Results & evaluation
The experiments conducted on the Explainable Multimodal Emotion Recognition (EMER) task demonstrate the effectiveness of the proposed model. The paper provides a comprehensive evaluation against multiple baseline models, showcasing significant improvements in performance metrics. The ablation studies further validate the necessity of the proposed components, reinforcing the claims made regarding the importance of local acoustic dynamics and timestamp synchronization.
Reproducibility
The paper includes sufficient implementation details, including the architecture, optimization strategies, and dataset descriptions, which facilitate reproducibility. However, the absence of a publicly available code repository or demo URL limits the ability for other researchers to directly replicate the results.
Limitations
While the model shows promising results, it occasionally misclassifies ambiguous emotional states, particularly in sarcastic utterances. Additionally, the performance can degrade in low-SNR scenarios due to background noise interference. These limitations highlight areas for future improvement, particularly in enhancing robustness against challenging acoustic conditions.
Broader impact
The advancements presented in AcoustEmo have significant implications for applications in empathetic conversational agents, mental health monitoring, and human-computer interaction. By improving the accuracy of emotion recognition in multimodal contexts, the model can contribute to more socially aware AI systems, enhancing user experiences in various interactive settings. The paper presents AcoustEmo, a time-sensitive MLLM that significantly enhances open-vocabulary emotion reasoning by capturing local acoustic dynamics through a novel Utterance-Aware Acoustic Q-Former. This work is a meaningful contribution to the field of multimodal emotion recognition, addressing critical gaps in existing methodologies and demonstrating substantial technical advancements.
Author of this paper? Send a correction.