USE: A Unified Model for Universal Sound Separation and Extraction
Abstract
Sound separation (SS) and target sound extraction (TSE) are fundamental techniques for addressing complex acoustic scenarios. While existing SS methods struggle with determining the unknown number of sound sources, TSE approaches require precisely specified clues to achieve optimal performance. This paper proposes a unified framework that synergistically combines SS and TSE to overcome their individual limitations. Our architecture employs two complementary components: 1) An Encoder-Decoder Attractor (EDA) network that automatically infers both the source count and corresponding acoustic clues for SS, and 2) A multi-modal fusion network that precisely interprets diverse user-provided clues (acoustic, semantic, or visual) for TSE. Through joint training with cross-task consistency constraints, we establish a unified latent space that bridges both paradigms. During inference, the system adaptively operates in either fully autonomous SS mode or clue-driven TSE mode. Experiments demonstrate remarkable performance in both tasks, with notable improvements of 1.4 dB SDR improvement in SS compared to baseline and 86\% TSE accuracy.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel approach to unify sound separation and extraction tasks, demonstrating significant improvements in performance while addressing key limitations of existing methods. The integration of multi-modal clues and adaptive inference modes positions this work as a valuable contribution to the field of audio processing and machine learning.
Methodology
The proposed USE framework integrates sound separation (SS) and target sound extraction (TSE) into a unified model, addressing the limitations of existing methods that require predefined source counts or high-quality clues. The architecture employs an Encoder-Decoder Attractor (EDA) network for inferring source counts and a multi-modal fusion network for interpreting various user-provided clues. The joint training strategy with cross-task consistency constraints is innovative, allowing the model to adaptively switch between SS and TSE modes during inference. However, the complexity of the architecture may pose challenges in practical applications, especially in real-time scenarios.
Results & evaluation
The experiments conducted on universal sound datasets demonstrate significant improvements in both SS and TSE tasks, with a reported 1.4 dB SDR improvement in SS and 86% accuracy in TSE. The use of diverse datasets, including the AudioSet and custom 2Mix and 3Mix datasets, strengthens the evaluation. However, the paper lacks detailed comparisons with a broader range of existing models, which could provide a clearer context for the reported improvements.
Reproducibility
The paper outlines the training and evaluation strategies, including the two-stage training process and the datasets used. However, the absence of a publicly available code repository limits reproducibility. The methodology is described in detail, but without access to the actual implementation, it may be challenging for others to replicate the results.
Limitations
The performance of the USE model is noted to be influenced by the cleanliness of the training data, which may affect its adaptability to various tasks. Additionally, the reliance on multiple modalities for clues could complicate its deployment in scenarios where such data is not readily available. The paper also does not address the computational efficiency of the model in real-time applications, which is critical for practical use.
Broader impact
The USE framework has the potential to significantly enhance audio processing applications, particularly in environments with complex sound mixtures, such as surveillance, multimedia content creation, and assistive technologies for the hearing impaired. By effectively separating and extracting sounds from various sources, it could lead to advancements in audio analysis and understanding, fostering further research in the field. The paper presents a novel approach to unify sound separation and extraction tasks, demonstrating significant improvements in performance while addressing key limitations of existing methods. The integration of multi-modal clues and adaptive inference modes positions this work as a valuable contribution to the field of audio processing and machine learning.
Author of this paper? Send a correction.