MAGE: Modality-Agnostic Music Generation and Editing
Abstract
Multimodal music creation requires models that can both generate audio from high-level cues and edit existing mixtures in a targeted manner. Yet most multimodal music systems are built for a single task and a fixed prompting interface, making their conditioning brittle when guidance is ambiguous, temporally misaligned, or partially missing. Common additive fusion or feature concatenation further weakens cross-modal grounding, often causing prompt drift and spurious musical content during generation and editing. We propose MAGE, a modality-agnostic framework that unifies multimodal music generation and mixture-grounded editing within a single continuous latent formulation. At its core, MAGE uses a Controlled Multimodal FluxFormer, a flow-based Transformer that learns controllable latent trajectories for synthesis and editing under any available subset of conditions. To improve grounding, we introduce Audio-Visual Nexus Alignment to select temporally consistent visual evidence for the audio timeline, and a cross-gated modulation mechanism that applies multiplicative control from aligned visual and textual cues to the audio latents, suppressing unsupported components rather than injecting them. Finally, we train with a dynamic modality-masking curriculum that exposes the model to text-only, visual-only, joint multimodal, and mixture-guided settings, enabling robust inference under missing modalities without training separate models. Experiments on the MUSIC benchmark show that MAGE supports effective multimodal-guided music generation and targeted editing, achieving competitive quality while offering a lightweight and flexible interface tailored to practical music workflows.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MAGE, a modality-agnostic framework for multimodal music generation and editing that effectively combines various conditioning inputs within a single continuous latent formulation. This work significantly advances the field of audio processing by addressing the complexities of multimodal music creation and providing a flexible, unified approach that enhances both generation and editing capabilities.
Methodology
The methodology presented in MAGE is robust and innovative, employing a Controlled Multimodal FluxFormer that integrates audio generation and editing in a unified framework. The use of Audio-Visual Nexus Alignment and cross-gated modulation enhances cross-modal grounding, addressing significant challenges in multimodal music generation. The dynamic modality-masking curriculum is a noteworthy approach that allows the model to adapt to various input conditions without requiring separate training for each modality, which is a significant advancement in the field.
Results & evaluation
The experiments conducted on the MUSIC benchmark are comprehensive and well-structured, demonstrating the effectiveness of MAGE in both multimodal-guided music generation and targeted editing. The evaluation metrics used, including SDR, SIR, SAR, FAD, and CLAP scores, provide a thorough assessment of the model's performance across different tasks, ensuring that both signal fidelity and semantic alignment are addressed.
Reproducibility
The paper provides detailed implementation information, including the architecture of the model, training procedures, and the datasets used. However, the absence of a publicly available code repository or demo limits the reproducibility of the results, as other researchers cannot easily validate or build upon the work.
Limitations
While the proposed framework is promising, it may still struggle with highly complex mixtures where the separation of overlapping sources is particularly challenging. Additionally, the reliance on specific datasets like MUSIC may limit the generalizability of the findings to other music genres or contexts.
Broader impact
The implications of MAGE extend beyond academic research, potentially influencing practical applications in music production, interactive audio editing, and creative tools for musicians. The ability to generate and edit music based on multimodal inputs could revolutionize how music is created and manipulated, making it more accessible to non-experts. The main contribution of this paper is the introduction of MAGE, a modality-agnostic framework for multimodal music generation and editing that effectively combines various conditioning inputs within a single continuous latent formulation. This work significantly advances the field of audio processing by addressing the complexities of multimodal music creation and providing a flexible, unified approach that enhances both generation and editing capabilities.
Author of this paper? Send a correction.