SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model
Abstract
A 4D audio-visual scene comprises a video, the dynamic geometry it depicts, and the sound sources that populate it, each with its own position and trajectory. Rendering such a scene from a novel viewpoint requires that every source be available as an individual waveform, so that it can be localized in the scene and propagated to the observer before the signals are mixed. Joint audio-video generators can synthesize the video and its soundtrack, but the soundtrack is emitted as a single audio-mix in which the sources are not individually accessible. We present SepGen, which extends a pretrained audio-video generator to emit the video, the mixed soundtrack, and one waveform per captioned source in one joint sampling run. SepGen supports two complementary modes: generation and separation. In generation mode, each source caption specifies what its stem contains. A two-speaker dialogue, for example, comes out as one stem per speaker in the original turn order. In separation mode, the input audio-mix remains clean while the captions specify what to extract, so the model can decompose a recording from a free-text description. We evaluate generation on scenes synthesized from text, and separation on scenes rendered by other generators and on real recordings of speech, music, and sound effects. Given an audio-mix and captions that carry the spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech, and it keeps the lead when the lines are removed from the captions. Code, checkpoints, and datasets are available at https://sepgen.github.io/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SepGen introduces a unified framework for multi-stem audio-video generation and separation by extending a pretrained joint generator with a protected audio-mix channel and block-diagonal caption routing, enabling the extraction of individual source waveforms necessary for 4D spatial audio rendering. The technical contribution is robust, leveraging a two-stage training strategy and an "Estimated Separation" inference schedule to ensure stems remain faithful to their captions while maintaining the integrity of the original audio-mix, effectively solving the problem of inaccessible source separation in joint audio-video synthesis.
Methodology
The paper proposes SepGen, a framework that extends a pretrained joint audio-video generator (specifically LTX-2.5) to output individual source stems alongside the video and mixed audio. The core technical contribution is a parameter-efficient adaptation using LoRA on the audio stream, where stems are treated as additional channels. Key innovations include a "protected" audio-mix channel that bypasses the adapter to preserve the backbone's original output quality, block-diagonal caption routing to ensure stems attend only to their specific captions, and a two-stage training strategy (separation first, then generation) that allows a single checkpoint to handle both tasks by varying the noise level of the audio-mix. The "Estimated Separation" inference schedule, where stems attend to the denoised estimate of the audio-mix rather than the noisy latent, is a clever mechanism to improve consistency during generation.
Results & evaluation
The evaluation covers both generation and separation modes. For separation, the model is compared against state-of-the-art language-queried separators (AudioSep, FlowSep, SAM Audio) on real recordings and cross-generator data. SepGen outperforms these baselines, particularly in speech separation, and maintains performance even when specific spoken lines are removed from captions. For generation, the paper evaluates the quality of the generated stems and their alignment with the video. The inclusion of a human A/B study adds qualitative depth. The use of a small dataset (4,843 examples) is a limitation, but the reliance on a strong pretrained backbone mitigates this.
Reproducibility
The authors provide code, checkpoints, and datasets at the project URL. The detailed description of the training schedule, attention masks, and LoRA gating mechanisms in the method section suggests high reproducibility. The specific hyperparameters (e.g., threshold $\tau_g=0.97$, probability 0.3 for clean mix in Stage 2) are explicitly stated.
Limitations
The model is currently limited to $N=2$ sources. The training dataset is relatively small (4,843 examples), which may limit generalization to complex scenes with many overlapping sources. The method relies heavily on the quality of the pretrained backbone (LTX-2.5); if the backbone's audio quality is poor, the stems will inherit these limitations. The paper does not extensively discuss computational overhead compared to the base model, though LoRA suggests it is minimal.
Broader impact
This work is significant for the field of 4D audio-visual scene rendering. By providing individual source waveforms, it enables spatial audio rendering from novel viewpoints, which is currently impossible with standard joint generators that only output a mixed soundtrack. It bridges the gap between generative video models and spatial audio applications, potentially impacting virtual reality, film production, and interactive media. SepGen introduces a unified framework for multi-stem audio-video generation and separation by extending a pretrained joint generator with a protected audio-mix channel and block-diagonal caption routing, enabling the extraction of individual source waveforms necessary for 4D spatial audio rendering. The technical contribution is robust, leveraging a two-stage training strategy and an "Estimated Separation" inference schedule to ensure stems remain faithful to their captions while maintaining the integrity of the original audio-mix, effectively solving the problem of inaccessible source separation in joint audio-video synthesis.
Author of this paper? Send a correction.