MusicLM: Generating Music From Text
Abstract
We introduce MusicLM, a model generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff". MusicLM casts the process of conditional music generation as a hierarchical sequence-to-sequence modeling task, and it generates music at 24 kHz that remains consistent over several minutes. Our experiments show that MusicLM outperforms previous systems both in audio quality and adherence to the text description. Moreover, we demonstrate that MusicLM can be conditioned on both text and a melody in that it can transform whistled and hummed melodies according to the style described in a text caption. To support future research, we publicly release MusicCaps, a dataset composed of 5.5k music-text pairs, with rich text descriptions provided by human experts.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MusicLM introduces a groundbreaking approach to generating high-fidelity music from text descriptions, significantly advancing the field of audio generation. The combination of innovative methodology, comprehensive experimental evaluation, and the introduction of a new dataset positions this work as a pivotal contribution to the intersection of machine learning and music.
Methodology
The methodology presented in MusicLM is robust, leveraging a hierarchical sequence-to-sequence modeling approach to generate high-fidelity music from text descriptions. The model integrates multi-stage autoregressive modeling, which is a significant advancement in audio generation, particularly in maintaining long-term coherence and audio quality. The introduction of a joint embedding model (MuLan) to bridge music and text descriptions is innovative, allowing for a more flexible and scalable training process without the need for extensive paired datasets. Additionally, the ability to condition on both text and melody adds a layer of complexity and versatility to the model.
Results & evaluation
The experiments conducted are thorough, comparing MusicLM against established baselines (Mubert and Riffusion) using both quantitative metrics (FAD, KLD, MCC) and qualitative human evaluations. The results demonstrate a clear advantage in audio quality and adherence to text descriptions, with detailed statistical analyses supporting the findings. The introduction of the MusicCaps dataset, specifically curated for this task, enhances the evaluation framework and provides a valuable resource for future research.
Reproducibility
The authors provide sufficient implementation details, including the architecture of the model, training processes, and evaluation metrics, which facilitate reproducibility. The public release of the MusicCaps dataset further supports this goal, allowing other researchers to validate and build upon the findings.
Limitations
While the model shows impressive capabilities, it struggles with complex text descriptions involving negations and precise temporal ordering. These limitations suggest areas for improvement in future iterations of the model. Additionally, the reliance on the quality of the training data raises concerns about biases and cultural representation in the generated music.
Broader impact
MusicLM has the potential to revolutionize music generation by providing tools that assist in creative processes, enabling users to generate music tailored to specific descriptions. However, it also raises ethical concerns regarding cultural appropriation and the risks of misappropriating creative content. The findings underscore the importance of responsible model development and the need for ongoing discussions about the implications of AI in creative fields. MusicLM introduces a groundbreaking approach to generating high-fidelity music from text descriptions, significantly advancing the field of audio generation. The combination of innovative methodology, comprehensive experimental evaluation, and the introduction of a new dataset positions this work as a pivotal contribution to the intersection of machine learning and music.
Curator note
Agostinelli et al., Google; text-conditional music generation; MuLan embeddings; raised music gen quality bar
Discussion
Talks & videos
Related later work
- MusicGen / AudioCraft 2023
- AudioLDM 2 2023
- YuE 2025
- Stable Audio (Stability AI) 2023
Author of this paper? Send a correction.