SAM Audio: Segment Anything in Audio
Abstract
General audio source separation is a key capability for multimodal AI systems that can perceive and reason about sound. Despite substantial progress in recent years, existing separation models are either domain-specific, designed for fixed categories such as speech or music, or limited in controllability, supporting only a single prompting modality such as text. In this work, we present SAM Audio, a foundation model for general audio separation that unifies text, visual, and temporal span prompting within a single framework. Built on a diffusion transformer architecture, SAM Audio is trained with flow matching on large-scale audio data spanning speech, music, and general sounds, and can flexibly separate target sources described by language, visual masks, or temporal spans. The model achieves state-of-the-art performance across a diverse suite of benchmarks, including general sound, speech, music, and musical instrument separation in both in-the-wild and professionally produced audios, substantially outperforming prior general-purpose and specialized systems. Furthermore, we introduce a new real-world separation benchmark with human-labeled multimodal prompts and a reference-free evaluation model that correlates strongly with human judgment.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SAM Audio, a foundation model for general audio separation that integrates multiple prompting modalities within a diffusion transformer architecture. This work represents a significant advancement in audio source separation, showcasing innovative methodologies and robust experimental results that could influence future research in multimodal AI systems.
Methodology
The methodology presented in SAM Audio is noteworthy for its integration of multiple prompting modalities (text, visual, and temporal) within a single framework. The use of a diffusion transformer architecture is innovative, particularly in the context of audio source separation, which has traditionally relied on more conventional neural network architectures. The flow matching training approach is also a significant contribution, as it allows for the model to be trained effectively on large-scale audio datasets. However, the paper could benefit from a more detailed explanation of the training process and the specific design choices made in the architecture.
Results & evaluation
The experimental evaluation is robust, with the authors demonstrating state-of-the-art performance across a diverse set of benchmarks. The introduction of a new real-world separation benchmark with human-labeled multimodal prompts is a strong point, as it enhances the relevance of the evaluation metrics. The results indicate substantial improvements over both general-purpose and specialized systems, which speaks to the effectiveness of the proposed model. However, additional details on the datasets used and the specific metrics for evaluation would enhance the credibility of the results.
Reproducibility
The paper lacks sufficient details regarding the implementation of the model, which raises concerns about reproducibility. While the results are promising, the absence of a publicly available code repository or detailed algorithmic descriptions makes it difficult for other researchers to replicate the findings or build upon this work.
Limitations
One limitation of the study is the potential overfitting to the specific datasets used for training and evaluation, which could affect the generalizability of the model. Additionally, while the model supports multiple prompting modalities, the paper does not thoroughly explore the limitations of each modality or the conditions under which the model may fail to perform optimally.
Broader impact
The implications of SAM Audio are significant, particularly in the context of multimodal AI systems. The ability to separate audio sources based on various prompts can enhance applications in fields such as music production, film editing, and assistive technologies for the hearing impaired. The model's versatility could lead to advancements in how machines understand and interact with audio in a more human-like manner. The main contribution of this paper is the introduction of SAM Audio, a foundation model for general audio separation that integrates multiple prompting modalities within a diffusion transformer architecture. This work represents a significant advancement in audio source separation, showcasing innovative methodologies and robust experimental results that could influence future research in multimodal AI systems.
Author of this paper? Send a correction.