Neural Audio Codecs for Prompt-Driven Universal Sound Separation
Abstract
Text-guided sound separation supports flexible audio editing across media and assistive applications, but existing models like AudioSep are too compute-heavy for edge deployment. Neural audio codec (NAC) models such as CodecFormer and SDCodec are compute-efficient but limited to fixed-class separation. We introduce CodecSep, the first NAC-based model for on-device universal, text-driven separation. CodecSep combines DAC compression with a Transformer masker modulated by CLAP-derived FiLM parameters. Across six open-domain benchmarks under matched training/prompt protocols, \textbf{CodecSep} surpasses \textbf{AudioSep} in separation fidelity (SI-SDR) while remaining competitive in perceptual quality (ViSQOL) and matching or exceeding fixed-stem baselines (TDANet, CodecFormer, SDCodec). In code-stream deployments, it needs just 1.35~GMACs end-to-end -- approximately $54\times$ less compute ($25\times$ architecture-only) than spectrogram-domain separators like AudioSep -- while remaining fully bitstream-compatible.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of CodecSep, a novel NAC-based model for universal sound separation that combines efficient audio processing with text-driven control, outperforming existing methods in both fidelity and computational efficiency. This work represents a significant advancement in the field of audio processing, particularly for applications requiring real-time performance on resource-constrained devices.
Methodology
The paper introduces CodecSep, a novel approach to universal sound separation that leverages neural audio codecs (NACs) and a transformer-based masker modulated by text embeddings. This combination allows for efficient, on-device sound separation while maintaining high fidelity and perceptual quality. The use of Feature-wise Linear Modulation (FiLM) to condition the transformer on text embeddings is an innovative aspect that enhances the model's ability to interpret prompts semantically. The methodology is well-structured, with a clear rationale for the design choices, particularly the decision to operate in the latent space of the codec rather than the spectrogram domain. This choice significantly reduces computational requirements and improves separation performance.
Results & evaluation
The authors conduct a comprehensive evaluation across multiple datasets, including both in-domain and open-domain benchmarks. The results demonstrate that CodecSep outperforms existing models like AudioSep in separation fidelity (measured by SI-SDR) while remaining competitive in perceptual quality (ViSQOL). The experiments are well-designed, with careful attention to matched training and prompt protocols, and the inclusion of ablation studies helps to isolate the contributions of different components of the model. The reported performance gains are substantial, indicating the effectiveness of the proposed approach.
Reproducibility
The paper mentions that supplementary code will be provided to facilitate reproducibility, which is a positive aspect. However, specific implementation details, hyperparameters, and training configurations are deferred to the appendix, which may pose challenges for full reproducibility unless the supplementary materials are made readily accessible.
Limitations
The paper acknowledges several limitations, including the modest scale of training data and prompt diversity, which could affect generalization. Additionally, it notes that while the model is robust to synonymic paraphrases, it has not been tested with prompts that include explicit temporal structures. The perceptual quality of sound effects (SFX) in some cases trails behind the best competing scores, indicating room for improvement.
Broader impact
The ability to perform efficient, text-guided sound separation has significant implications for various applications, including media production, assistive technologies, and real-time audio editing. The model's efficiency makes it suitable for deployment on edge devices, which could democratize access to advanced audio processing capabilities. The main contribution of this paper is the introduction of CodecSep, a novel NAC-based model for universal sound separation that combines efficient audio processing with text-driven control, outperforming existing methods in both fidelity and computational efficiency. This work represents a significant advancement in the field of audio processing, particularly for applications requiring real-time performance on resource-constrained devices.
Author of this paper? Send a correction.