A Mixture-of-Experts Model for Multimodal Emotion Recognition in Conversations
Abstract
Emotion Recognition in Conversations (ERC) presents unique challenges, requiring models to capture the temporal flow of multi-turn dialogues and to effectively integrate cues from multiple modalities. We propose Mixture of Speech-Text Experts for Recognition of Emotions (MiSTER-E), a modular Mixture-of-Experts (MoE) framework designed to decouple two core challenges in ERC: modality-specific context modeling and multimodal information fusion. MiSTER-E leverages large language models (LLMs) fine-tuned for both speech and text to provide rich utterance-level embeddings, which are then enhanced through a convolutional-recurrent context modeling layer. The system integrates predictions from three experts-speech-only, text-only, and cross-modal-using a learned gating mechanism that dynamically weighs their outputs. To further encourage consistency and alignment across modalities, we introduce a supervised contrastive loss between paired speech-text representations and a KL-divergence-based regulariza-tion across expert predictions. Importantly, MiSTER-E does not rely on speaker identity at any stage. Experiments on three benchmark datasets-IEMOCAP, MELD, and MOSI-show that our proposal achieves 70.9%, 69.5%, and 87.9% weighted F1-scores respectively, outperforming several baseline speech-text ERC systems. We also provide various ablations to highlight the contributions made in the proposed approach.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the MiSTER-E framework, which effectively addresses the challenges of multimodal emotion recognition in conversations through a novel Mixture-of-Experts approach. This work represents a meaningful advancement in the field, combining innovative methodology with strong experimental validation, although it could improve in terms of reproducibility and practical application.
Methodology
The proposed MiSTER-E framework is innovative in its use of a Mixture-of-Experts (MoE) architecture to tackle the dual challenges of modality-specific context modeling and multimodal information fusion in Emotion Recognition in Conversations (ERC). The integration of large language models (LLMs) for generating utterance-level embeddings is a significant strength, as it allows for rich contextual representations. The gating mechanism to dynamically weigh the outputs from different experts (speech-only, text-only, and cross-modal) is a novel approach that enhances the model's adaptability. The introduction of a supervised contrastive loss and KL-divergence regularization further strengthens the model's ability to align multimodal representations, which is crucial for effective emotion recognition.
Results & evaluation
The experiments conducted on three benchmark datasets (IEMOCAP, MELD, and MOSI) provide a solid foundation for evaluating the model's performance. The reported F1-scores demonstrate that MiSTER-E outperforms several baseline systems, indicating its effectiveness. However, the paper could benefit from a more detailed comparison with state-of-the-art methods and an analysis of the significance of the improvements achieved. The ablation studies are a positive addition, as they help clarify the contributions of various components of the proposed approach.
Reproducibility
The paper lacks detailed implementation information, such as hyperparameters, training procedures, and specific configurations of the LLMs used. This omission may hinder reproducibility. Including a supplementary material section with this information or providing a link to a code repository would greatly enhance the reproducibility of the results.
Limitations
One limitation of the study is the reliance on benchmark datasets, which may not fully capture the complexities of real-world conversations. Additionally, the model's performance in diverse contexts or with varying speaker identities is not addressed, which could affect its generalizability. The absence of a demo or project URL also limits accessibility for further exploration of the model.
Broader impact
The implications of this research are significant, particularly in applications such as virtual assistants, customer service bots, and mental health monitoring systems, where understanding emotional cues in conversations is crucial. The model's ability to integrate multimodal information could lead to more empathetic and context-aware AI systems, enhancing user experience and interaction quality. The main contribution of this paper is the MiSTER-E framework, which effectively addresses the challenges of multimodal emotion recognition in conversations through a novel Mixture-of-Experts approach. This work represents a meaningful advancement in the field, combining innovative methodology with strong experimental validation, although it could improve in terms of reproducibility and practical application.
Author of this paper? Send a correction.