YuE: Scaling Open Foundation Models for Long-Form Music Generation
Abstract
We tackle the task of long-form music generation--particularly the challenging \textbf{lyrics-to-song} problem--by introducing YuE, a family of open foundation models based on the LLaMA2 architecture. Specifically, YuE scales to trillions of tokens and generates up to five minutes of music while maintaining lyrical alignment, coherent musical structure, and engaging vocal melodies with appropriate accompaniment. It achieves this through (1) track-decoupled next-token prediction to overcome dense mixture signals, (2) structural progressive conditioning for long-context lyrical alignment, and (3) a multitask, multiphase pre-training recipe to converge and generalize. In addition, we redesign the in-context learning technique for music generation, enabling versatile style transfer (e.g., converting Japanese city pop into an English rap while preserving the original accompaniment) and bidirectional generation. Through extensive evaluation, we demonstrate that YuE matches or even surpasses some of the proprietary systems in musicality and vocal agility. In addition, fine-tuning YuE enables additional controls and enhanced support for tail languages. Furthermore, beyond generation, we show that YuE's learned representations can perform well on music understanding tasks, where the results of YuE match or exceed state-of-the-art methods on the MARBLE benchmark. Keywords: lyrics2song, song generation, long-form, foundation model, music generation
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of YuE, an open foundation model for long-form music generation that effectively addresses the lyrics-to-song problem, achieving competitive performance with proprietary systems while maintaining a focus on accessibility and reproducibility. The comprehensive methodology and rigorous evaluation underscore its significance in advancing the field of AI-driven music generation.
Methodology
The methodology presented in this paper is robust and innovative, employing a dual-token strategy for track-decoupled next-token prediction, which effectively addresses the challenges of generating coherent long-form music. The structural progressive conditioning and redesigned in-context learning techniques further enhance the model's ability to maintain lyrical alignment and musical coherence across extended durations. The use of a multitask, multiphase pre-training approach is particularly noteworthy, as it integrates various auxiliary tasks to improve the model's performance on the primary task of lyrics-to-song generation.
Results & evaluation
The experiments conducted are extensive and well-structured, including both human evaluations and automatic metrics to assess the model's performance against proprietary systems. The results indicate that YuE performs competitively, achieving strong musicality and vocal agility, while also demonstrating the ability to generate longer audio outputs than existing models. The use of diverse datasets and comprehensive evaluation metrics enhances the credibility of the findings.
Reproducibility
The paper provides a detailed account of the training setup, data sources, and evaluation protocols, which supports reproducibility. The authors have made their code and demo available, further facilitating replication of their work. However, the complexity of the model and the extensive training data may pose challenges for full reproduction without significant computational resources.
Limitations
One limitation noted is the model's performance in vocal and accompaniment acoustic quality, which could be improved with better audio tokenization methods. Additionally, the reliance on specific datasets may limit the generalizability of the results across different musical styles and languages. The paper also acknowledges potential copyright concerns with the generated content, which is an important consideration in music generation research.
Broader impact
The implications of this work are significant, as it democratizes access to high-quality music generation tools through an open-source framework. This could foster innovation in music composition, education, and therapy, making music creation more accessible to a wider audience. The model's ability to handle multilingual lyrics and style transfer also suggests potential applications in diverse cultural contexts. The main contribution of this paper is the introduction of YuE, an open foundation model for long-form music generation that effectively addresses the lyrics-to-song problem, achieving competitive performance with proprietary systems while maintaining a focus on accessibility and reproducibility. The comprehensive methodology and rigorous evaluation underscore its significance in advancing the field of AI-driven music generation.
Curator note
Yuan et al.; large-scale music LM with lyrics conditioning; open-source music generation at scale
Discussion
Talks & videos
Author of this paper? Send a correction.