Stemphonic: All-at-once Flexible Multi-stem Music Generation
Abstract
Music stem generation, the task of producing musically-synchronized and isolated instrument audio clips, offers the potential of greater user control and better alignment with musician workflows compared to conventional text-to-music models. Existing stem generation approaches, however, either rely on fixed architectures that output a predefined set of stems in parallel, or generate only one stem at a time, resulting in slow inference despite flexibility in stem combination. We propose Stemphonic, a diffusion-/flow-based framework that overcomes this trade-off and generates a variable set of synchronized stems in one inference pass. During training, we treat each stem as a batch element, group synchronized stems in a batch, and apply a shared noise latent to each group. At inference-time, we use a shared initial noise latent and stem-specific text inputs to generate synchronized multi-stem outputs in one pass. We further expand our approach to enable one-pass conditional multi-stem generation and stem-wise activity controls to empower users to iteratively generate and orchestrate the temporal layering of a mix. We benchmark our results on multiple open-source stem evaluation sets and show that Stemphonic produces higher-quality outputs while accelerating the full mix generation process by 25 to 50%. Demos at: https://stemphonic-demo.vercel.app.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Stemphonic, a novel framework for efficient multi-stem music generation, significantly advancing the field of audio generation through innovative methodologies and promising experimental results.
Methodology
The methodology presents a novel framework that integrates diffusion and flow-based models for music stem generation, addressing the limitations of existing approaches by allowing for variable and synchronized stem outputs in a single inference pass. The introduction of techniques such as stem grouping and noise sharing during training is particularly innovative, as it enhances inter-stem cohesion and synchronization, which are critical in music generation tasks. The approach is well-structured and builds upon established generative models, showcasing a clear progression from theory to practical application.
Results & evaluation
The experiments are comprehensive, utilizing multiple datasets and evaluation metrics to assess the quality of generated stems and mixes. The results demonstrate significant improvements in generation quality and efficiency, with quantitative metrics such as Fréchet Audio Distance (FAD) providing a robust framework for evaluation. The ablation studies effectively highlight the contributions of the proposed techniques, reinforcing the validity of the claims made by the authors.
Reproducibility
The paper provides detailed implementation specifics, including architecture choices, training procedures, and dataset descriptions, which facilitate reproducibility. However, the absence of a publicly available code repository limits the ease with which other researchers can replicate the results.
Limitations
One limitation is the reliance on specific datasets for training and evaluation, which may not fully capture the diversity of music styles and genres. Additionally, while the model shows promise in generating synchronized stems, the quality of generated audio may still vary depending on the complexity of the input prompts and conditions.
Broader impact
The proposed framework has significant implications for music production, enabling greater creative control for musicians and content creators. By facilitating the generation of isolated instrument tracks, it can streamline workflows in music composition and production, potentially democratizing music creation for non-experts. The ability to generate stems on-demand could also enhance collaborative efforts in music-making. The paper introduces Stemphonic, a novel framework for efficient multi-stem music generation, significantly advancing the field of audio generation through innovative methodologies and promising experimental results.
Author of this paper? Send a correction.