AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow

Duojia Li, Shuhan Zhang, Zihan Qian ...Full author list in the original paper.

Xiamen University +8 · arXiv

Affiliations: Xiamen University, Hong Kong SAR, Nanjing University, School of Artificial Intelligence, School of Electronic Science and Engineering, School of Informatics, School of Intelligence Science and Technology, Shenzhen Loop Area Institute, The Chinese University of Hong Kong

Paper

Abstract

In target speaker extraction (TSE), we aim to recover target speech from a multi-talker mixture using a short enrollment utterance as reference. Recent studies on diffusion and flow-matching generators have improved target-speech fidelity. However, multi-step sampling increases latency, and one-step solutions often rely on a mixture-dependent time coordinate that can be unreliable for real-world conversations. We present AlphaFlowTSE, a one-step conditional generative model trained with a Jacobian-vector product (JVP)-free AlphaFlow objective. AlphaFlowTSE learns mean-velocity transport along a mixture-to-target trajectory starting from the observed mixture, eliminating auxiliary mixing-ratio prediction, and stabilizes training by combining flow matching with an interval-consistency teacher-student target. Experiments on Libri2Mix and REAL-T confirm that AlphaFlowTSE improves target-speaker similarity and real-mixture generalization for downstream automatic speech recognition (ASR).

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The main contribution of this paper is the introduction of AlphaFlowTSE, a novel one-step generative model for target speaker extraction that significantly improves extraction quality and generalization while maintaining low latency. This work represents a meaningful advancement in the field of audio processing, particularly in enhancing the fidelity and efficiency of speaker extraction systems.

Methodology

The methodology proposed in AlphaFlowTSE is innovative, as it introduces a one-step generative model for target speaker extraction that utilizes a Jacobian-vector product (JVP)-free AlphaFlow objective. The combination of trajectory matching and interval-consistency teacher-student supervision is a significant advancement in the field, addressing the challenges of latency and accuracy in real-world applications. The use of mean-velocity transport in the complex STFT domain is particularly noteworthy, as it aligns training with inference, making the model more efficient.

Results & evaluation

The experiments conducted on the Libri2Mix and REAL-T datasets are comprehensive and demonstrate the effectiveness of AlphaFlowTSE. The results show improvements in target-speaker similarity and generalization to real conversational mixtures, with strong performance metrics such as PESQ, ESTOI, and SI-SDR. The ablation studies regarding the MR predictor further validate the robustness of the proposed model.

Reproducibility

The paper provides detailed implementation details, including training protocols, model architecture, and evaluation metrics, which contribute to reproducibility. However, the lack of a public repository or demo URL limits the ease of access for other researchers to replicate the results.

Limitations

One limitation is the reliance on the enrollment utterance, which may not always be available in practical scenarios. Additionally, while the model shows strong performance in controlled environments, its effectiveness in highly variable real-world conditions remains to be fully validated.

Broader impact

The advancements in target speaker extraction have significant implications for applications in personal communication systems, such as virtual assistants and conference call technologies. The ability to accurately extract a target speaker's voice in real-time can enhance user experiences in various audio processing tasks, including automatic speech recognition and noise suppression. The main contribution of this paper is the introduction of AlphaFlowTSE, a novel one-step generative model for target speaker extraction that significantly improves extraction quality and generalization while maintaining low latency. This work represents a meaningful advancement in the field of audio processing, particularly in enhancing the fidelity and efficiency of speaker extraction systems.

Author of this paper? Send a correction.