FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation
Abstract
Language-queried audio source separation (LASS) aims to extract target sources from audio mixtures according to natural language descriptions, offering a flexible and scalable interface for audio source separation. However, most existing LASS methods rely on discriminative, mask-based models, which estimate masks from the input mixture. These methods often over-suppress target sounds or fail to fully separate them, especially when multiple sound events strongly overlap in complex acoustic scenes. In this work, we propose FlowSep2, a text-conditioned flow-matching generative model for LASS. Instead of directly predicting a separation mask, FlowSep2 learns to generate the target source representation from Gaussian noise in a latent space, conditioned on both the mixture representation and the text query. Specifically, we employ rectified flow matching with a Diffusion Transformer backbone. We further incorporate Self-Flow, a self-supervised flow-matching paradigm, into our LASS framework. By encouraging semantically structured latent representations under the generative objective, Self-Flow improves the model's ability to separate target sources according to text queries. Experiments on multiple LASS benchmarks show that FlowSep2 achieves state-of-the-art performance and demonstrates enhanced sound separation results in challenging scenarios with overlapping sound events.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
FlowSep2 establishes a strong generative baseline for language-queried audio source separation by effectively combining Rectified Flow Matching with self-supervised semantic representation alignment, achieving state-of-the-art results on multiple benchmarks and demonstrating robust generalization to complex acoustic scenes.
Methodology
The paper proposes FlowSep2, a language-queried audio source separation (LASS) system based on Rectified Flow Matching (RFM) with a Diffusion Transformer (DiT) backbone. The core methodological contribution is the integration of "Self-Flow," a self-supervised representation alignment technique adapted from Self-Flow (SRFM), into the generative separation framework. This involves a teacher-student EMA setup where the student predicts a noisy view with mixed flow times, while the teacher provides a cleaner, uniform view, enforcing semantic consistency in the latent space. The approach replaces traditional mask-based discriminative models with a generative paradigm that synthesizes the target source directly from noise, conditioned on mixture audio and text. The use of RFM allows for efficient sampling compared to standard DDPMs. The methodology is technically sound and builds logically on recent advances in flow matching and self-supervised representation learning.
Results & evaluation
The authors evaluate FlowSep2 on six benchmarks: AudioCaps, VGGSound, ESC-50, MUSDB18, and DCASE 2024 Task 9 (Synth and Real). They employ a comprehensive suite of metrics including FAD, CLAP Score, CLAP$_A$ Score, AudioBox Aesthetics, and SAJ (SAM Audio Judge), supplemented by human subjective evaluations. The results demonstrate state-of-the-art performance on most general LASS benchmarks (AudioCaps, VGGSound, DCASE), significantly outperforming previous discriminative and generative baselines like LASS-Net, AudioSep, and the original FlowSep. Notably, on MUSDB18, FlowSep2 performs competitively in a zero-shot setting despite being trained only on general sound data, whereas SAM-Audio (trained on proprietary music data) leads. The ablation studies effectively isolate the contributions of the DiT backbone, RFM vs. DDPM, and the Self-Flow alignment strategy, showing consistent improvements.
Reproducibility
The paper provides detailed descriptions of the model architecture, training hyperparameters (learning rate, batch size, optimizer), and dataset construction. The use of standard components (FLAN-T5, Stable-Audio VAE) aids reproducibility. However, the code repository is not explicitly linked in the provided text (only a demo URL is given), and the exact preprocessing steps for the large-scale training data (5,650 hours) are summarized but not fully detailed in the snippet. The inclusion of ablation studies and specific metric values enhances the ability to verify claims.
Limitations
A significant limitation is the reliance on generative modeling, which can introduce artifacts or "hallucinations" not present in the target source, although the CLAP$_A$ score suggests good fidelity. The model's performance on music separation (MUSDB18) is inferior to specialized models like SAM-Audio, likely due to the lack of music-specific training data. The computational cost of training and inference for DiT-based flow models is substantial, which may limit accessibility. Additionally, the "Self-Flow" adaptation requires careful tuning of the teacher-student update rates and flow time sampling strategies.
Broader impact
This work advances the field of audio source separation by demonstrating the efficacy of generative flow models for language-guided tasks. It offers a more flexible interface for audio editing and retrieval, allowing users to specify targets via natural language. The improved separation capabilities have applications in multimedia content creation, assistive listening technologies, and audio forensics. The open-source nature of the evaluation (via demo) and the use of public benchmarks contribute to the community's ability to benchmark future systems. FlowSep2 establishes a strong generative baseline for language-queried audio source separation by effectively combining Rectified Flow Matching with self-supervised semantic representation alignment, achieving state-of-the-art results on multiple benchmarks and demonstrating robust generalization to complex acoustic scenes.
Author of this paper? Send a correction.