UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE
Abstract
Recent advances in unified multimodal models indicate a clear trend towards comprehensive content generation. However, the auditory domain remains a significant challenge, with music and speech often developed in isolation, hindering progress towards universal audio synthesis. This separation stems from inherent task conflicts and severe data imbalances, which impede the development of a truly unified audio generation model. To address this challenge, we propose UniMoE-Audio, a unified speech and music generation model within a novel Dynamic-Capacity Mixture-of-Experts (MoE) framework. Architecturally, UniMoE-Audio introduces a Top-P routing strategy for dynamic expert number allocation, and a hybrid expert design comprising routed experts for domain-specific knowledge, shared experts for domain-agnostic features, and null experts for adaptive computation skipping. To tackle data imbalance, we introduce a three-stage training curriculum: 1) Independent Specialist Training leverages original datasets to instill domain-specific knowledge into each "proto-expert" without interference; 2) MoE Integration and Warmup incorporates these specialists into the UniMoE-Audio architecture, warming up the gate module and shared expert using a subset of balanced dataset; and 3) Synergistic Joint Training trains the entire model end-to-end on the fully balanced dataset, fostering enhanced cross-domain synergy. Extensive experiments show that UniMoE-Audio not only achieves state-of-the-art performance on major speech and music generation benchmarks, but also demonstrates superior synergistic learning, mitigating the performance degradation typically seen in naive joint training. Our findings highlight the substantial potential of specialized MoE architecture and curated training strategies in advancing the field of universal audio generation. Homepage: https://mukioxun.github.io/Uni-MoE-site/home.html
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of UniMoE-Audio, a novel framework that effectively integrates speech and music generation through a Dynamic-Capacity Mixture-of-Experts architecture, demonstrating state-of-the-art performance and addressing critical challenges in the field. The technical contributions, particularly the innovative routing strategy and training curriculum, position this work as a significant advancement in universal audio generation.
Methodology
The proposed methodology of UniMoE-Audio is innovative, utilizing a Dynamic-Capacity Mixture-of-Experts (MoE) framework that effectively addresses the challenges of data imbalance and task conflicts in audio generation. The introduction of a Top-P routing strategy for dynamic expert allocation is particularly noteworthy, as it allows the model to adaptively select the number of experts based on the input, enhancing efficiency and performance. The three-stage training curriculum is well-structured, ensuring that domain-specific knowledge is effectively integrated into the model while also promoting cross-domain synergy. However, the complexity of the architecture may pose challenges in terms of implementation and understanding.
Results & evaluation
The experiments conducted demonstrate a thorough evaluation of the model against major benchmarks in speech and music generation. The results indicate that UniMoE-Audio achieves state-of-the-art performance, showcasing its effectiveness in both domains. The paper provides sufficient details on the datasets used and the evaluation metrics, which strengthens the credibility of the findings. However, it would benefit from additional comparative analysis with other recent models to further contextualize its performance.
Reproducibility
The paper lacks detailed implementation specifics, such as hyperparameter settings and training configurations, which are crucial for reproducibility. While the methodology is described, the absence of a publicly available code repository limits the ability of other researchers to replicate the results. The provided demo URL offers some insight into the model's capabilities, but a comprehensive code release would significantly enhance reproducibility.
Limitations
One limitation of the proposed model is its complexity, which may hinder practical deployment in real-world applications. Additionally, while the model addresses data imbalance, the effectiveness of the three-stage training curriculum in various real-world scenarios remains to be fully validated. The reliance on a balanced dataset in the latter training stages may not always be feasible, and the performance in highly imbalanced scenarios could be a concern.
Broader impact
The potential applications of UniMoE-Audio are significant, spanning various fields such as entertainment, education, and accessibility. The ability to generate high-quality speech and music in a unified framework could lead to advancements in audio synthesis technologies, enhancing user experiences in multimedia applications. Furthermore, the insights gained from this research could inform future studies on multimodal learning and audio generation. The main contribution of this paper is the introduction of UniMoE-Audio, a novel framework that effectively integrates speech and music generation through a Dynamic-Capacity Mixture-of-Experts architecture, demonstrating state-of-the-art performance and addressing critical challenges in the field. The technical contributions, particularly the innovative routing strategy and training curriculum, position this work as a significant advancement in universal audio generation.
Author of this paper? Send a correction.