MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
Abstract
We present MoST (Mixture of Speech and Text), a novel multimodal large language model that seamlessly integrates speech and text processing through our proposed Modality-Aware Mixture of Experts (MAMoE) architecture. While current multimodal models typically process diverse modality representations with identical parameters, disregarding their inherent representational differences, we introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type. MAMoE simultaneously enhances modality-specific learning and cross-modal understanding through two complementary components: modality-specific expert groups that capture domain-specific patterns and shared experts that facilitate information transfer between modalities. Building on this architecture, we develop an efficient transformation pipeline that adapts the pretrained MoE language model through strategic post-training on ASR and TTS datasets, followed by fine-tuning with a carefully curated speech-text instruction dataset. A key feature of this pipeline is that it relies exclusively on fully accessible, open-source datasets to achieve strong performance and data efficiency. Comprehensive evaluations across ASR, TTS, audio language modeling, and spoken question answering benchmarks show that MoST consistently outperforms existing models of comparable parameter counts. Our ablation studies confirm that the modality-specific routing mechanism and shared experts design significantly contribute to performance gains across all tested domains. To our knowledge, MoST represents the first fully open-source speech-text LLM built on a Mixture of Experts architecture. \footnote{We release MoST model, training code, inference code, and training data at https://github.com/NUS-HPC-AI-Lab/MoST
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MoST represents a significant advancement in multimodal large language models by integrating speech and text through a novel architecture that enhances both modality-specific learning and cross-modal understanding. The comprehensive evaluation and innovative methodology position this work as a meaningful contribution to the field of machine learning, particularly in multimodal AI applications.
Methodology
The paper introduces MoST, a novel architecture that employs a Modality-Aware Mixture of Experts (MAMoE) to effectively integrate speech and text modalities. The methodology is well-structured, with a clear delineation of the routing mechanisms that allow for specialized processing of different modalities. The proposed training pipeline is innovative, leveraging open-source datasets for post-training and fine-tuning, which enhances the model's adaptability and efficiency. The dual-output capability (text and audio) is a significant advancement, allowing for a wide range of applications in speech-text tasks.
Results & evaluation
The experiments are comprehensive, covering various benchmarks for ASR, TTS, audio language modeling, and spoken question answering. The evaluations demonstrate that MoST outperforms existing models with comparable parameter counts, providing strong empirical evidence of its effectiveness. The ablation studies further validate the contributions of the MAMoE architecture, highlighting the importance of modality-specific routing and shared experts.
Reproducibility
The authors emphasize reproducibility by releasing the model weights, training code, and datasets. They provide detailed descriptions of the training process, hyperparameters, and model configurations, which are crucial for replicating the results. The inclusion of a reproducibility statement strengthens the credibility of the research.
Limitations
While the model shows impressive results, the paper acknowledges potential societal risks, such as voice spoofing, which could arise from the technology's capabilities. Additionally, the reliance on open-source datasets may limit the model's performance in scenarios requiring proprietary or specialized data. Future work could explore more sophisticated expert initialization strategies to optimize the model further.
Broader impact
MoST has significant implications for enhancing human-computer interaction through improved speech and text integration, potentially benefiting applications in accessibility, education, and conversational AI. However, ethical considerations regarding misuse and privacy must be addressed as the technology advances. MoST represents a significant advancement in multimodal large language models by integrating speech and text through a novel architecture that enhances both modality-specific learning and cross-modal understanding. The comprehensive evaluation and innovative methodology position this work as a meaningful contribution to the field of machine learning, particularly in multimodal AI applications.
Author of this paper? Send a correction.