LiveSynth: A Streaming Neural Synthesizer for Instrument Cloning and Text-to-Instrument
Abstract
Recently, generative neural synthesizers have enabled zero-shot instrument cloning from reference recordings and timbre generation from text prompts. However, existing systems generate complete audio segments offline, precluding their use as playable instruments. We present LiveSynth, a streaming neural synthesizer with flexible timbre control and interactive generation capabilities, including zero-shot instrument cloning and text-to-instrument synthesis. A feedback-free causal Transformer conditioned on MIDI events and a CLAP timbre embedding outputs continuous audio latents at 100 Hz, which a causal variational autoencoder decodes into audio. Frame-wise timbre conditioning enables continuous timbre morphing, while MIDI-conditioning dropout enables autonomous continuation from a performed prefix. On the NSynth and Slakh datasets, LiveSynth achieved timbre similarity and spectral reconstruction quality comparable to offline baselines. On an M3 MacBook Air, LiveSynth required 8.8 ms of computation per 10 ms audio frame (an RTF of 0.88) with 10 ms algorithmic latency, enabling real-time interaction with a performer.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
LiveSynth introduces a streaming neural synthesizer capable of real-time zero-shot instrument cloning and text-to-instrument synthesis with 10 ms latency. By leveraging a feedback-free causal Transformer and frame-wise timbre conditioning, it achieves competitive quality with offline baselines while enabling interactive features like continuous timbre morphing and autonomous performance continuation, marking a significant step toward playable generative audio models.
Methodology
The paper proposes LiveSynth, a streaming neural synthesizer that adapts the feedback-free causal Transformer framework from LiveBand for instrument cloning and text-to-instrument synthesis. The core methodological contribution is the integration of frame-wise timbre conditioning (using CLAP embeddings) with a causal VAE decoder operating at 100 Hz. The use of MIDI-conditioning dropout to enable autonomous continuation from a performed prefix is a clever training trick that facilitates interactive "call-and-response" scenarios. The decision to use continuous latents rather than quantized tokens is well-justified by the need for low latency and high reconstruction quality, avoiding the error accumulation and quality degradation associated with autoregressive token generation in strict causal settings. The Procrustes alignment for text-to-audio embedding mapping is a practical and effective solution to the modality gap in CLAP embeddings.
Results & evaluation
The experimental evaluation is rigorous, comparing LiveSynth against strong offline baselines (TokenSynth, CTD) on both NSynth and Slakh datasets. The metrics used (MERT, PANNs, MSS, F1, CE, PQ) provide a comprehensive view of timbre similarity, spectral quality, and perceptual aesthetics. The real-time inference benchmark on an M3 MacBook Air is highly relevant and convincing, demonstrating an RTF of 0.88 with 10 ms latency, which is critical for the "playable instrument" claim. The ablation on timbre morphing and autonomous continuation provides good evidence for the specific interactive capabilities. However, the lack of human listening tests (MOS/ABX) is a significant gap for a paper claiming perceptual quality improvements.
Reproducibility
The paper provides detailed architectural specifications (parameter counts, hop sizes, attention windows) and training hyperparameters. The code and demo are available via the provided URL. The use of standard datasets (NSynth, Slakh) and open-source components (CLAP, Vocos, DAC) enhances reproducibility. However, the specific implementation of the Procrustes alignment and the exact configuration of the adversarial training might require careful attention to replicate exactly.
Limitations
The primary limitation is the lack of subjective human evaluation (listening tests), which is standard for audio generation papers. The paper acknowledges lower note accuracy (F1) and production quality (PQ) compared to some offline baselines, which is a trade-off for real-time performance. The rhythmic dependence in autonomous continuation is weak, as noted by the high IOI-JSD distance. The system is currently limited to mono audio and specific instrument types found in the training data.
Broader impact
This work bridges the gap between offline generative audio models and real-time interactive music tools. It enables new forms of human-AI musical interaction, such as live jamming with AI-generated instruments that can morph timbres or continue performances autonomously. This has implications for music production, education, and performance art. The streaming architecture could be extended to other domains requiring real-time conditional generation. LiveSynth introduces a streaming neural synthesizer capable of real-time zero-shot instrument cloning and text-to-instrument synthesis with 10 ms latency. By leveraging a feedback-free causal Transformer and frame-wise timbre conditioning, it achieves competitive quality with offline baselines while enabling interactive features like continuous timbre morphing and autonomous performance continuation, marking a significant step toward playable generative audio models.
Author of this paper? Send a correction.