SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling
Abstract
This paper addresses the challenge of audio-visual single-microphone speech separation and enhancement in the presence of real-world environmental noise. Our approach is based on generative inverse sampling, where we model clean speech and ambient noise with dedicated diffusion priors and jointly leverage them to recover all underlying sources. To achieve this, we reformulate a recent inverse sampler to match our setting. We evaluate on mixtures of 1, 2, and 3 speakers with noise and show that, despite being entirely unsupervised, our method consistently outperforms leading supervised baselines in \ac{WER} across all conditions. We further extend our framework to handle off-screen speaker separation. Moreover, the high fidelity of the separated noise component makes it suitable for downstream acoustic scene detection. Demo page: https://ssnapsicml.github.io/ssnapsicml2026/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel unsupervised generative method for audio-visual speech separation that significantly advances the field. The technical contributions, particularly in leveraging diffusion models and visual cues, offer promising directions for future research and practical applications in speech processing.
Methodology
The methodology proposed in SSNAPS is innovative, leveraging generative inverse sampling with diffusion models to separate speech from background noise in an unsupervised manner. The paper reformulates existing inverse sampling techniques to accommodate multiple independent signals and integrates visual cues from lip movements to enhance separation accuracy. The approach's flexibility in handling varying numbers of speakers and the introduction of a novel loss function for off-screen speaker separation are significant advancements. However, the reliance on visual data may limit applicability in scenarios without such cues.
Results & evaluation
The experimental evaluation is robust, with comprehensive testing on mixtures of 1, 2, and 3 speakers across different noise conditions. The results demonstrate that SSNAPS consistently outperforms leading supervised baselines in terms of word error rate (WER), showcasing the effectiveness of the unsupervised approach. The paper provides detailed metrics and comparisons, enhancing the credibility of the findings. However, the paper could benefit from additional qualitative assessments or user studies to further validate the audio quality of the separated signals.
Reproducibility
The paper includes sufficient implementation details, including datasets, model architectures, and hyperparameters, which facilitate reproducibility. The authors provide a demo page, but the absence of a public code repository limits the ability for others to reproduce the results independently. The detailed explanation of the experimental setup and evaluation metrics is commendable, yet sharing the actual code would enhance transparency.
Limitations
One key limitation is the dependency on visual data for performance, which may not be available in all real-world applications. Additionally, while the method shows promise in separating speech from noise, the computational efficiency could be improved, as indicated by the longer runtime compared to supervised methods. The paper also does not address potential challenges in scenarios with more complex noise environments or overlapping speech characteristics.
Broader impact
The advancements presented in this paper have significant implications for various applications, including telecommunication, assistive technologies for the hearing impaired, and audio-visual media processing. By improving speech separation in noisy environments, the method could enhance user experiences in real-world settings, making communication clearer and more effective. The unsupervised nature of the approach also suggests potential for broader adoption in diverse applications without the need for extensive labeled datasets. The paper presents a novel unsupervised generative method for audio-visual speech separation that significantly advances the field. The technical contributions, particularly in leveraging diffusion models and visual cues, offer promising directions for future research and practical applications in speech processing.
Author of this paper? Send a correction.