Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators
Abstract
Interactive streaming music generation promises the use of generative models for live performance and co-creation that is impossible with offline models. However, SOTA models exist in the discrete-AR regime, requiring industrial levels of compute for both training and inference. In this work, we investigate whether audio diffusion models, with their wide support in the open-source community but non-streaming bidirectional nature, can be repurposed efficiently into interactive models accessible on consumer hardware. By taking a critical look at the modern pipeline for block-wise outpainting diffusion, we identify critical inefficiencies during inference that result in strictly worse computational efficiency than their discrete-AR counterparts. We propose Live Music Diffusion Models (LMDMs), a simple modification of the generative diffusion process that recovers, and then outperforms, the inference complexity of the discrete Live Music Models (LMMs) through block-wise KV Caching. Unlike LMMs, LMDMs further enable stable post-training alignment through our novel ARC-Forcing paradigm, reducing error accumulation without any explicit RL or reward models. We demonstrate the application of LMDMs in a number of creative domains, including text-conditioned generation, sketch-based music synthesis, and jamming. We finally show how LMDMs can be used as a generative instrument in a real artist-AI collaboration, utilizing LMDMs as a "generative delay" to transform musicians' improvisation live for variable timbral effects while running locally on a consumer gaming laptop.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a novel approach to interactive music generation through the introduction of Live Music Diffusion Models, which enhance the efficiency and applicability of diffusion models for real-time performance. The methodology is innovative, addressing key challenges in the field, and the experimental results demonstrate significant technical contributions that could impact both research and practical applications in music technology.
Methodology
The paper introduces Live Music Diffusion Models (LMDMs), which modify existing diffusion models to enable efficient block-wise KV caching and a novel post-training method called ARC-Forcing. The methodology effectively addresses the computational inefficiencies of traditional autoregressive models by leveraging the strengths of diffusion models while maintaining real-time performance suitable for consumer hardware. The proposed routing mechanism and attention masking techniques are well-justified and demonstrate a clear understanding of the limitations of prior models.
Results & evaluation
The experiments are comprehensive, comparing LMDMs against state-of-the-art models in various creative domains, including text-conditioned generation and live musician interaction. The use of multiple evaluation metrics (FD, KL, CLAP) provides a robust framework for assessing model performance. The results indicate that LMDMs achieve competitive quality and significantly reduced latency, highlighting their practical applicability in real-world scenarios.
Reproducibility
The paper provides sufficient details regarding the training and inference setup, including model parameters and datasets used. However, the absence of a publicly available code repository may hinder full reproducibility. The authors do provide a demo URL, which partially mitigates this concern by allowing users to experience the model's capabilities firsthand.
Limitations
The authors acknowledge limitations regarding genre bias in training data and the model's responsiveness to text features compared to clean audio content. Additionally, the need for further latency reduction is identified as a critical area for future work, particularly for achieving seamless real-time interaction.
Broader impact
The development of LMDMs represents a significant advancement in interactive music generation, potentially transforming live performance and co-creation experiences. By making high-quality generative models accessible on consumer hardware, this work could democratize music creation and inspire new forms of artistic collaboration between musicians and AI. This paper presents a novel approach to interactive music generation through the introduction of Live Music Diffusion Models, which enhance the efficiency and applicability of diffusion models for real-time performance. The methodology is innovative, addressing key challenges in the field, and the experimental results demonstrate significant technical contributions that could impact both research and practical applications in music technology.
Author of this paper? Send a correction.