DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation
Abstract
Text and lyrics specify broad musical characteristics and sung content but offer limited control over musical timing, melody, and reference-based style. We introduce DiffSynth-Music (https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music), a framework that adds composable audio conditioning to a music synthesis backbone through layer-wise key-value injection. The three template models, Control, Prosody, and Reference, are initialized from the backbone diffusion transformer and trained with conditional flow matching. They support five control types: beats, vocals, accompaniment, prosody, and reference audio. A shared variational autoencoder maps conditioning waveforms into a common latent space, enabling their attention memories to be combined. With the template timestep fixed at the clean-data endpoint and other inputs held constant, each control cache is computed once and reused throughout sampling. Training pairs are derived from music recordings using beat extraction, source separation, vocal resynthesis, and reference-excerpt selection. Single-control evaluations on Mandarin and English songs demonstrate improved adherence across all five control types and better lyric fidelity under vocal conditioning relative to the backbone. Automatic music-quality and instruction-following scores remain broadly comparable to those of the evaluated base models, with metric-specific trade-offs. We release the three template models to support research and creative applications in controllable music generation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper makes a strong commitment to reproducibility by open-sourcing the three template models (Control, Prosody, Reference). This is a significant contribution to the research community. The methodology section provides sufficient detail on the architecture, KV injection, and the construction of the five audio control signals, including specific tools used (Essentia, Demucs, pYIN). This level of detail allows researchers to understand and potentially replicate the control signal generation process. However, a limitation for full reproducibility is the use of a private dataset of approximately 60k music recordings for training and a private 50-sample Mandarin dataset for testing. While the English test set uses MUSDB18-HQ, the full training data and the Mandarin test data are not publicly available. This means that while the *models* are released, replicating the *training process* or *full evaluation* exactly would be challenging without access to similar proprietary datasets. The specific parameters for training (e.g., learning rates, batch sizes, number
Methodology
The paper introduces DiffSynth-Music, a framework for controllable music generation built upon a frozen Diffusion Transformer (DiT) backbone (ACE-Step-1.5-XL-SFT). The core methodology involves injecting layer-wise key-value (KV) representations derived from audio conditioning signals into the generation branch. This approach is conceptually related to adapter-based methods like ControlNet but is tailored for audio and uses a shared VAE to map diverse conditioning waveforms into a common latent space. A key strength is the design of three template models (Control, Prosody, Reference) that support five distinct control types: beats, vocals, accompaniment, prosody, and reference audio. The detailed construction of these control signals from raw music recordings (e.g., beat extraction, source separation, vocal resynthesis for prosody, loudest excerpt for reference) is well-described and practical. The prosody generation method, which uses a sinusoidal carrier with estimated pitch and envelope to reduce phonetic/timbral cues, is a clever way to achieve melodic control without enforcing specific vocal identity. The KV-injection mechanism, where control keys and values are concatenated with the generation branch's keys and values, allows for composable conditioning. The efficiency gain from computing each control cache once at a fixed template timestep (clean-data endpoint) and reusing it throughout sampling is a significant practical advantage, reducing computational overhead. The training formulation uses conditional flow matching, which is a suitable choice for learning continuous-time vector fields, and the adapter-only optimization ensures the pretrained backbone remains frozen, preserving its general generation capabilities. The overall architecture is sound, well-motivated, and addresses a clear need in controllable music generation.
Results & evaluation
The experimental evaluation is comprehensive and well-structured. The authors evaluate DiffSynth-Music on both Mandarin and English songs, demonstrating its applicability across different languages, which is a strong point. The comparison against several state-of-the-art music generation systems (ACE-Step-1.5-XL-SFT, DiffRhythm-2, HeartMuLa-3B, MiniMax-Music3, LeVo-2-Large) provides a robust baseline. Critically, the backbone model (ACE-Step-1.5-XL-SFT) is included as a baseline to specifically highlight the improvements gained by adding the audio-conditioned templates. The evaluation metrics are appropriate and cover both control adherence and general music quality. For control adherence, metrics like Beat-F1, Cemgil (beats), V-MSE, PER (vocals), A-MSE (accompaniment), Pitch_50 (prosody), and MuLan-A (reference) are used. The results clearly show that DiffSynth-Music significantly outperforms all baselines across all seven control adherence metrics, often by a large margin (e.g., Beat-F1 from ~0.3 to ~0.85, Pitch_50 from ~0.04 to ~0.47). This strongly validates the effectiveness of the proposed conditioning framework. To assess general music quality, AudioBox-Aesthetics scores (CE, CU, PC, PQ) and MuLan-T (text-music alignment) are used. The results indicate that DiffSynth-Music maintains "broadly comparable" scores to the evaluated base models, with some metric-specific trade-offs. While some scores are slightly lower than the backbone, the overall message is that the added controllability does not severely degrade the general generation quality, which is a reasonable and expected outcome for such an adapter-based system. The experimental setup and results convincingly support the paper's claims.
Reproducibility
The paper makes a strong commitment to reproducibility by open-sourcing the three template models (Control, Prosody, Reference). This is a significant contribution to the research community. The methodology section provides sufficient detail on the architecture, KV injection, and the construction of the five audio control signals, including specific tools used (Essentia, Demucs, pYIN). This level of detail allows researchers to understand and potentially replicate the control signal generation process. However, a limitation for full reproducibility is the use of a private dataset of approximately 60k music recordings for training and a private 50-sample Mandarin dataset for testing. While the English test set uses MUSDB18-HQ, the full training data and the Mandarin test data are not publicly available. This means that while the *models* are released, replicating the *training process* or *full evaluation* exactly would be challenging without access to similar proprietary datasets. The specific parameters for training (e.g., learning rates, batch sizes, number
Author of this paper? Send a correction.