Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes
Abstract
Streaming audio-visual target speaker extraction (AVTSE) is essential for latency-sensitive applications. However, current causal systems face two coupled gaps. First, standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes. Second, high-performance causal architectures are too heavy for edge deployment. Lightweight alternatives try to reclaim capacity by repeatedly invoking a small separator, which offsets the savings. To address the first gap, we release RealSSA, a realistic AVTSE benchmark consisting of RealSSA-Sim and RealSSA-Real. RealSSA-Sim provides controllable simulated scenes with 1--5 dynamic near-field interfering speakers; scenes with far-field interference additionally include 10--15 far-field interfering speakers. RealSSA-Real provides a held-out real-recorded evaluation set. To address the second gap, we propose Falcon, a causal AVTSE method. Its Separator performs multi-scale time-frequency separation in a single encoder--decoder pass. We train Falcon with an encoder-space speaker contrastive loss. This loss suppresses near-field leakage at zero inference cost and transfers across backbones. Falcon achieves state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. Compared with the causal AV-TFGridNet baseline, it cuts parameters by 91.9%, computation by 20.2 times, and GPU inference latency by 11.1 times. Code and demo are available at https://cslikai.cn/Falcon/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Falcon, a lightweight causal AVTSE model, and RealSSA, a realistic benchmark, achieving significant efficiency gains over existing causal baselines while maintaining state-of-the-art extraction quality. The technical contribution is strong, particularly in addressing the efficiency-quality trade-off in causal systems and providing a more realistic evaluation environment for the field.
Methodology
The paper proposes Falcon, a causal audio-visual target speaker extraction (AVTSE) model, and RealSSA, a new benchmark. The methodology addresses two specific gaps: the lack of realistic acoustic conditions in existing benchmarks and the high computational cost of causal models. Falcon utilizes a single-pass asymmetric U-Net separator with multi-scale time-frequency modeling, avoiding the iterative recurrence found in prior lightweight models like SwiftNet. A key technical contribution is the "encoder-space speaker contrastive loss," which explicitly suppresses leakage from near-field interferers during training without adding inference cost. The architecture uses a frozen FRCNN visual backbone and a causal LSTM for video processing, fused with audio features via addition. The design is logically sound, aiming to balance latency constraints with extraction quality by replacing iterative small separators with a single, more efficient pass.
Results & evaluation
The experimental setup is robust, evaluating Falcon on the new RealSSA-Sim, RealSSA-Real, and standard benchmarks (LRS2-2Mix, LRS3-2Mix, VoxCeleb2-2Mix). The paper claims state-of-the-art performance on these benchmarks. Specifically, it reports a 91.9% reduction in parameters and an 11.1x reduction in GPU inference latency compared to the causal AV-TFGridNet baseline. The inclusion of a real-world recorded set (RealSSA-Real) is a significant strength, as it tests generalization beyond simulated mixtures. The use of metrics like SI-SNR, WER, DNSMOS, and NISQA provides a comprehensive evaluation of both separation quality and perceptual/recognizability aspects.
Reproducibility
The paper provides a code and demo URL (https://cslikai.cn/Falcon/), which enhances reproducibility. The dataset generation pipeline for RealSSA-Sim is detailed, including specific parameters for room impulse responses (gpuRIR), source placement, and mixing levels (EBU R128). The architectural details of the Separator, including convolution types, padding strategies, and the definition of the causal layer normalization, are described in sufficient detail for re-implementation.
Limitations
The visual backbone (FRCNN) is frozen, which may limit the model's ability to adapt to diverse visual conditions or lighting changes not present in the training data. The contrastive loss relies on having clean references for interferers during training, which is feasible in simulation but might be challenging in unsupervised or semi-supervised real-world settings. The paper is an arXiv preprint with 0 citations, so long-term impact is yet to be determined. The "RealSSA-Real" set is described as having pixelated faces, which is good for privacy but may introduce artifacts that affect visual feature extraction.
Broader impact
This work has significant implications for edge deployment of speech enhancement systems, such as in hearing aids, video conferencing, and smart speakers. By providing a lightweight causal model and a realistic benchmark, it facilitates the development of more practical, low-latency audio-visual systems. The benchmark addresses a critical need for evaluating models under realistic, non-anechoic conditions with varying numbers of speakers. The paper introduces Falcon, a lightweight causal AVTSE model, and RealSSA, a realistic benchmark, achieving significant efficiency gains over existing causal baselines while maintaining state-of-the-art extraction quality. The technical contribution is strong, particularly in addressing the efficiency-quality trade-off in causal systems and providing a more realistic evaluation environment for the field.
Author of this paper? Send a correction.