Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extraction
Abstract
Online audio-visual target-speaker extraction aims to remove competing voices while preserving speech quality and bounding lookahead. Existing extractors are built and evaluated for separation on synthetic mixtures, leaving listening quality and meeting behavior largely untested. We introduce Audio-Visual Personalized Voice Quality Enhancement (AV-PVQE), which approaches these requirements from the other direction. We start from a personalized speech enhancement model that reconstructs a requested voice at high quality but confuses the target in 46% of two-speaker mixtures despite clean enrollment. Adding mouth features at its speaker-conditioning input and jointly fine-tuning the visual and reconstruction networks reduces this rate to 1.6%, with no future frames and 20 ms of algorithmic delay. Compared with an online autoregressive audio-visual extractor, AV-PVQE yields separation gains on two synthetic benchmarks and larger gains on recorded meetings, and keeps its advantage on excerpts with more speakers than the fine-tuning mixtures. In personalized P.835 listening tests on two meeting corpora, it improves overall quality over this extractor by 0.57 and 0.63 MOS, with similar mean rating relative to the starting model. Preservation and rejection tests show that it keeps the target intact when no competing voice is present and suppresses competing speech when the target is absent.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a robust adaptation of a personalized speech enhancement model for online audio-visual target-speaker extraction, demonstrating significant improvements in target recovery and listening quality over existing extractors on both synthetic and real-world meeting data. By leveraging the prior knowledge of a deployed enhancer and integrating visual cues through a gated conditioning mechanism, the method effectively addresses the selection challenge while maintaining high speech quality, supported by extensive objective and subjective evaluations that highlight its practical deployability and the trade-offs involved in such adaptation.
Methodology
The paper proposes a pragmatic and effective adaptation of a deployed personalized speech enhancement model (PVQE) into an online audio-visual target-speaker extraction system (V+E). The core methodological contribution is the integration of visual cues (mouth motion via AV-HuBERT) into the existing speaker-conditioning path of the enhancer, rather than training a separator from scratch. The use of a gated residual connection to fuse enrollment embeddings with visual features is a sound architectural choice that allows the model to retain its enhancement capabilities while learning selection. The fine-tuning strategy, which involves removing lookahead operations and rearranging decoder filters for causal processing, is well-described and technically rigorous. The approach effectively leverages the prior knowledge of the enhancement model to solve the extraction problem, addressing the "selection" gap in personalized enhancement.
Results & evaluation
The experimental evaluation is extensive and high-quality. The authors test on both synthetic mixtures (LRS3, VoxCeleb2) and, crucially, recorded meeting corpora (AMI, MCoRec, MTM, UniTalk), which is a significant strength as it moves beyond the standard synthetic benchmark limitations. The inclusion of a large-scale human listening test (170 listeners, P.835 protocol) provides strong evidence for the perceptual quality claims. The results show consistent improvements over the baseline extractor (AVASE) in both objective metrics (SI-SNRi, WAcc) and subjective quality (MOS). The preservation and rejection tests further validate the system's deployability, showing it does not distort the target when no interference is present. The analysis of the UniTalk results, where background noise suppression degrades, is honest and provides valuable insight into the trade-offs of this adaptation strategy.
Reproducibility
The paper provides sufficient detail for reproducibility, including model architecture dimensions, training data sources, and fine-tuning procedures. The use of standard public datasets (LRS3, VoxCeleb2, AMI, MCoRec) aids reproducibility, although the internal MTM dataset is not publicly available. The specific hyperparameters and initialization strategies (e.g., zero-initialization of the enrollment projection) are clearly stated. However, the code and model weights are not explicitly linked in the provided text, which is a minor drawback for immediate reproducibility.
Limitations
The primary limitation is the degradation in background noise suppression on in-the-wild data (UniTalk) compared to the original enhancement model, indicating that the adaptation to extraction comes at a cost to general noise robustness. The system relies on the presence of synchronized video, which may not be available in all deployment scenarios. Additionally, the comparison with PVQE (the starting model) is somewhat confounded because PVQE is not designed for extraction, so the "quality retention" claim is relative to a model that fails at the primary task (selection) in many cases.
Broader impact
This work has significant practical impact for real-time communication systems (e.g., video conferencing). By demonstrating that a deployed enhancement model can be adapted for extraction with minimal architectural changes and high perceptual quality, it offers a viable path for improving user experience in noisy or multi-speaker environments. The focus on low-latency, online processing makes it highly relevant for industrial applications. The rigorous evaluation on recorded meetings sets a new standard for how such systems should be tested, encouraging the field to move beyond synthetic benchmarks. The paper presents a robust adaptation of a personalized speech enhancement model for online audio-visual target-speaker extraction, demonstrating significant improvements in target recovery and listening quality over existing extractors on both synthetic and real-world meeting data. By leveraging the prior knowledge of a deployed enhancer and integrating visual cues through a gated conditioning mechanism, the method effectively addresses the selection challenge while maintaining high speech quality, supported by extensive objective and subjective evaluations that highlight its practical deployability and the trade-offs involved in such adaptation.
Author of this paper? Send a correction.