PromptSep: Generative Audio Separation via Multimodal Prompting
Abstract
Recent breakthroughs in language-queried audio source separation (LASS) have shown that generative models can achieve higher separation audio quality than traditional masking-based approaches. However, two key limitations restrict their practical use: (1) users often require operations beyond separation, such as sound removal; and (2) relying solely on text prompts can be unintuitive for specifying sound sources. In this paper, we propose PromptSep to extend LASS into a broader framework for general-purpose sound separation. PromptSep leverages a conditional diffusion model enhanced with elaborated data simulation to enable both audio extraction and sound removal. To move beyond text-only queries, we incorporate vocal imitation as an additional and more intuitive conditioning modality for our model, by incorporating Sketch2Sound as a data augmentation strategy. Both objective and subjective evaluations on multiple benchmarks demonstrate that PromptSep achieves state-of-the-art performance in sound removal and vocal-imitation-guided source separation, while maintaining competitive results on language-queried source separation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
PromptSep offers a unified framework for sound extraction and removal that overcomes key limitations of existing LASS systems. The integration of vocal imitation as a query modality addresses the ambiguity and limitations of text prompts, offering a more intuitive interface for users. Through comprehensive evaluations, PromptSep demonstrates state-of-the-art performance in sound removal and vocal-imitation-guided separation, while remaining competitive in standard LASS settings.
Methodology
The methodology presented in PromptSep is innovative, as it combines a conditional diffusion model with multimodal prompting that includes both text and vocal imitation. This dual conditioning approach addresses the limitations of traditional language-queried audio source separation (LASS) systems, allowing for more intuitive user interactions and enabling both sound extraction and removal. The incorporation of a data simulation pipeline enhances the model's training process, allowing it to generalize better across various sound types and conditions. The use of Sketch2Sound for augmenting vocal imitation data is a notable strength, as it provides a richer dataset for training.
Results & evaluation
The experiments are well-structured, utilizing multiple benchmarks for both objective and subjective evaluations. The paper reports strong performance across various metrics, demonstrating the effectiveness of the proposed model in both sound extraction and removal tasks. The inclusion of both in-domain and out-of-domain datasets adds robustness to the evaluation, although the reliance on subjective evaluations could be further detailed in terms of participant demographics and scoring consistency.
Reproducibility
The paper provides a reasonable level of detail regarding the model architecture and training procedures, but it lacks specific implementation details that would enhance reproducibility, such as hyperparameter settings and training duration. The release of the VimSketchGen dataset is a positive step towards facilitating further research, but additional resources, such as code or pre-trained models, would be beneficial for the community.
Limitations
One limitation is the potential complexity of the model, which may require significant computational resources for training and inference. Additionally, while the model shows promise in handling vocal imitations, the paper does not explore the combined effects of using both text and vocal imitation during inference, which could limit its applicability in real-world scenarios. The subjective evaluation could also be influenced by the biases of the human annotators.
Broader impact
The advancements made in audio source separation through PromptSep have significant implications for various applications, including music production, film editing, and accessibility technologies for the hearing impaired. By allowing users to specify sounds more intuitively, the model could democratize audio editing tools, making them more accessible to non-experts. PromptSep offers a unified framework for sound extraction and removal that overcomes key limitations of existing LASS systems. The integration of vocal imitation as a query modality addresses the ambiguity and limitations of text prompts, offering a more intuitive interface for users. Through comprehensive evaluations, PromptSep demonstrates state-of-the-art performance in sound removal and vocal-imitation-guided separation, while remaining competitive in standard LASS settings.
Author of this paper? Send a correction.