AURA: Unified Multimodal Framework for Conversational Music Editing
Abstract
Instruction-guided music editors typically process each request independently, limiting their ability to support workflows in which users progressively refine a track. We introduce AURA, a unified multimodal framework for conversational music editing. AURA uses a multimodal large language model to interpret the complete dialogue history, an optional image, and reference audio, distilling the editing intent into compact concept tokens. A concept-to-audio module injects these tokens and frame-aligned reference features into a frozen MusicGen backbone, enabling precise edits while preserving unaffected content. AURA optimizes only 91M parameters while retaining 1.9B frozen backbone parameters. Experiments on Slakh2100 and MoisesDB demonstrate substantial improvements in edit correctness and content preservation over existing instruction-guided methods, including a 4-5 times reduction in FAD for out-of-domain addition and removal.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AURA introduces a unified multimodal framework for conversational music editing that uses compact concept tokens from a frozen MLLM to condition a frozen MusicGen backbone, achieving significant improvements in edit correctness and content preservation while maintaining high parameter efficiency.
Methodology
The paper proposes AURA, a framework that effectively decouples multimodal intent understanding from audio generation. The core innovation lies in the "Concept-to-Audio" (C2A) module and the use of compact concept tokens generated by a frozen Multimodal LLM (Qwen2.5-Omni). By projecting these tokens into the conditioning space of a frozen MusicGen backbone, the authors avoid the instability and high computational cost of fine-tuning large generative models. The use of a Bidirectional Feature Alignment Module (BiFAM) with FiLM modulation to inject frame-aligned reference audio features is a technically sound approach to preserving unedited content, as it allows the decoder to attend to the reference structure without appending tokens to the autoregressive sequence. The parameter efficiency (91M trainable vs 1.9B frozen) is a significant practical advantage.
Results & evaluation
The experiments are comprehensive, covering both in-domain (Slakh2100) and out-of-domain (MoisesDB) benchmarks. The metrics chosen are appropriate for this task, including FAD, CLAP, KL divergence, SSIM, and SI-SDR. The results show substantial improvements in content preservation (SSIM and SI-SDR) over baselines like M2UGen, Instruct-MusicGen, and LeVo. Specifically, the multi-turn editing experiments demonstrate that AURA handles conversational context better than single-turn baselines, which suffer from error accumulation. The ablation study confirms the utility of concept tokens over full-text conditioning. However, the paper lacks subjective human evaluation (MOS/ABX), which is standard in audio generation research to validate objective metrics.
Reproducibility
The paper provides sufficient architectural details, including the specific MLLM used, the number of concept tokens, LoRA ranks, and training hyperparameters. The use of standard open-source components (MusicGen, EnCodec, Qwen2.5-Omni) enhances reproducibility. However, the code repository is not explicitly linked in the text provided (only a demo page), which may hinder immediate reproduction.
Limitations
The primary limitation acknowledged is the performance in stem extraction tasks, where the generative approach lags behind dedicated separation models. Additionally, the reliance on a specific MLLM (Qwen2.5-Omni) may introduce biases or limitations inherent to that model's audio understanding capabilities. The lack of subjective evaluation is a notable gap.
Broader impact
This work contributes to the field of controllable audio generation by introducing a scalable, parameter-efficient method for conversational editing. It bridges the gap between high-level language understanding and low-level audio synthesis, offering a viable path for interactive music production tools. The framework's design could be extended to other audio editing tasks or modalities. AURA introduces a unified multimodal framework for conversational music editing that uses compact concept tokens from a frozen MLLM to condition a frozen MusicGen backbone, achieving significant improvements in edit correctness and content preservation while maintaining high parameter efficiency.
Author of this paper? Send a correction.