Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model
Abstract
While generative models have set new benchmarks for Target Speaker Extraction (TSE), their inherent reliance on global context precludes deployment in real-time applications. Direct adaptation to streaming scenarios often leads to catastrophic inference performance degradation due to the severe mismatch between training and streaming inference. To bridge this gap, we present the first autoregressive (AR) models tailored for streaming TSE. Our approach introduces a Chunk-wise Interleaved Splicing Paradigm that ensures highly efficient and stable streaming inference. To ensure the coherence between the extracted speech segments, we design a historical context refinement mechanism that mitigates boundary discontinuities by leveraging historical information. Experiments on Libri2Mix show that while AR generative baseline exhibits performance degradation at low latencies, our approach maintains 100% stability and superior intelligibility. Furthermore, our streaming results are comparable to or even surpass offline baselines. Additionally, our model achieves a Real-Time-Factor (RTF) of 0.248 on consumer-level GPUs. This work provides empirical evidence that AR generative backbones are viable for latency-sensitive applications through the Chunk-wise Interleaved Splicing Paradigm.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents the first autoregressive generative backbone tailored for streaming Target Speaker Extraction, filling a critical research void. The technical contributions, particularly the innovative Chunk-wise Interleaved Splicing Paradigm and historical context refinement mechanism, represent significant advancements in the field, with the potential to improve real-time audio processing applications substantially.
Methodology
The paper presents a novel autoregressive model specifically designed for streaming Target Speaker Extraction (TSE), introducing the Chunk-wise Interleaved Splicing Paradigm. This approach effectively addresses the mismatch between training and streaming inference by ensuring causality and stability in real-time applications. The historical context refinement mechanism is a significant addition that enhances the coherence of extracted speech segments, mitigating boundary discontinuities. The methodology is well-structured, with clear definitions and a logical flow from problem identification to proposed solutions. The use of autoregressive models in a streaming context is innovative, and the interleaved splicing paradigm is a clever engineering solution that maintains efficiency.
Results & evaluation
The experimental results are robust, showcasing a comprehensive evaluation against both generative and discriminative baselines. The use of the Libri2Mix dataset is appropriate, and the metrics employed (DNSMOS, NISQA, WER, etc.) are relevant for assessing speech quality and intelligibility. The results demonstrate that the proposed method not only maintains stability at low latencies but also achieves comparable or superior performance to offline models. The ablation studies provide valuable insights into the effectiveness of the historical context refinement and input strategies, reinforcing the contributions of the proposed methodology.
Reproducibility
The paper provides sufficient implementation details, including the architecture of the model, the training protocol, and the evaluation metrics. However, the lack of a public demo or project URL limits the reproducibility of the results. Future work could benefit from sharing code and models to facilitate further research and validation by the community.
Limitations
One limitation is the reliance on specific datasets, which may not fully capture the diversity of real-world scenarios. Additionally, while the proposed method shows promise, the performance at extreme low latencies (e.g., below 80ms) is not thoroughly evaluated. There may also be concerns regarding the generalizability of the model to other languages or dialects, which could affect its applicability in diverse settings.
Broader impact
This work has significant implications for real-time applications such as teleconferencing, voice-controlled systems, and multi-turn dialogue interactions. By enabling high-quality speech extraction in latency-sensitive environments, the proposed method can enhance user experiences in various audio processing applications. The approach could also inspire further research into autoregressive models for other real-time audio tasks, potentially leading to broader advancements in the field of speech processing. The paper presents the first autoregressive generative backbone tailored for streaming Target Speaker Extraction, filling a critical research void. The technical contributions, particularly the innovative Chunk-wise Interleaved Splicing Paradigm and historical context refinement mechanism, represent significant advancements in the field, with the potential to improve real-time audio processing applications substantially.
Author of this paper? Send a correction.