Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance
Abstract
Zero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present Anysynth, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that \tool outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semantic-acoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zero-shot instrument cloning. Demo audios are available at https://anysynth-demo.github.io/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a significant advancement in zero-shot instrument cloning by replacing information-bottleneck embeddings with in-context learning in a DiT architecture, supported by a novel Asymmetric CFG strategy that effectively decouples semantic and acoustic guidance.
Methodology
The paper proposes "Anysynth," an embedding-free neural synthesizer for zero-shot instrument cloning. The core methodological innovation lies in reformulating the task as an In-Context Learning (ICL) problem using a Diffusion Transformer (DiT) with Flow Matching. Instead of compressing reference audio into a fixed-length embedding (like CLAP), the model conditions directly on the uncompressed reference mel spectrogram and MIDI as a prefix. This allows the self-attention mechanism to dynamically attend to fine-grained acoustic details (transients, room reverb) from the reference. A key technical contribution is the "Asymmetric Hierarchical Classifier-Free Guidance" (CFG). The authors argue that standard symmetric CFG causes gradient conflicts between semantic (MIDI) and acoustic (timbre) conditions. By factorizing the probability $p(c_m, c_r | x_t)$ as $p(c_r | c_m, x_t)p(c_m | x_t)$, they decouple the guidance, applying strong structural guidance first and then refining with timbre. This is a theoretically sound and practically effective modification to standard diffusion/flow matching conditioning strategies. The MIDI encoding is also enhanced with a binary onset channel to better capture note boundaries, addressing a common failure mode in piano roll representations.
Results & evaluation
The experimental setup is robust, utilizing a mix of synthetic datasets (NSynth, Slakh) and real-world recordings (MAESTRO, GuitarSet). The evaluation metrics are comprehensive, including PANNs for timbre similarity, Onset F1 for melody adherence, MERT for perceptual quality, and Audiobox Aesthetics for overall quality. The results demonstrate that Anysynth outperforms embedding-based baselines (CLAP-conditioned DiT) and autoregressive/token-based baselines (TokenSynth, CTD). A significant finding is the "prompt-length scaling" property: unlike embedding-based models which saturate quickly, Anysynth's performance improves with longer reference prompts, validating the ICL approach. The ablation studies effectively isolate the benefits of in-context conditioning over CLAP embeddings and validate the superiority of Asymmetric CFG over Combined and Symmetric CFG variants. The UMAP visualizations provide qualitative support for the clustering of timbres.
Reproducibility
The paper provides detailed implementation specifics, including the DiT architecture (1024 hidden dim, 25 layers, 16 heads), training hyperparameters (batch size, learning rate, warmup), and dataset construction details (filtering rules for NSynth/Slakh). The use of standard components (Vocos vocoder, RoPE, AdaLN) aids reproducibility. However, the code repository is not explicitly linked in the text (only a demo page), which is a minor hurdle for immediate reproduction, though the architectural details are sufficient for a competent researcher to implement.
Limitations
The paper acknowledges that performance gains from prompt length diminish after 8 seconds, suggesting a limit to the utility of very long prompts for short generation tasks. The model is evaluated only on keyboard, guitar, and bass instruments; generalization to more complex orchestral instruments or vocals is not demonstrated. The reliance on a pre-trained vocoder (Vocos) means that the quality of the final waveform is partially dependent on the vocoder's capability, although this is standard practice. The computational cost of attending to long reference sequences in a DiT might be higher than embedding-based methods during inference, although the authors claim no major overhead due to parallel computation.
Broader impact
This work advances the field of controllable audio generation by providing a more faithful and flexible method for instrument cloning. It addresses a critical bottleneck in current systems (information loss in embeddings) and offers a scalable solution (ICL) that improves with more data/context. This has implications for music production tools, virtual instrument synthesis, and audio restoration. The Asymmetric CFG technique is generalizable to other multimodal generation tasks where semantic and stylistic/acoustic conditions are entangled. The paper presents a significant advancement in zero-shot instrument cloning by replacing information-bottleneck embeddings with in-context learning in a DiT architecture, supported by a novel Asymmetric CFG strategy that effectively decouples semantic and acoustic guidance.
Author of this paper? Send a correction.