PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction
Abstract
Training target speaker extraction (TSE) models for real conversational mixtures remains challenging because large-scale training corpora and clean target speech for supervision are unavailable. We present PS4, a proxy-supervised training framework for TSE in real conversational mixtures, with two main contributions. First, we construct a large-scale corpus of 71,771 training samples derived from four public datasets, covering both Chinese and English scenarios. Each sample contains an overlapping speech mixture, per-speaker enrollment audio, a ground-truth transcript, and frame-level voice activity labels. Second, we propose a proxy-supervised joint training strategy that fine-tunes a BSRNN-based TSE model using four complementary differentiable objectives: ASR cross-entropy, speaker similarity, frame-level voice activity detection, and perceptual audio quality. Starting from a publicly available pre-trained checkpoint, only the BSRNN separator is updated during fine-tuning. On the REAL-T challenge leaderboard, PS4 ranks 2nd overall, achieving the best speaker similarity and timing F1 among all submitted systems.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a practical and effective proxy-supervised training framework for target speaker extraction, successfully leveraging multiple auxiliary signals to achieve state-of-the-art performance on real-world conversational speech benchmarks.
Methodology
The paper proposes PS4, a proxy-supervised training framework for Target Speaker Extraction (TSE) in real conversational scenarios. The core methodological contribution is the design of four complementary differentiable loss functions to replace the unavailable clean target speech signal: ASR cross-entropy (using Whisper), speaker similarity (cosine margin ranking), frame-level Voice Activity Detection (VAD), and perceptual audio quality (DNSMOS). The approach leverages a pre-trained BSRNN-ECAPA model and fine-tunes only the separator. While the individual components (Whisper loss, speaker embedding losses) are known in the literature, their specific combination and application to the "real-world" TSE problem without clean references is a coherent and practical engineering contribution. The novelty lies in the systematic integration of these proxies to handle the domain gap between simulated and real data. EXPERIMENTAL_EVALUTION: The authors construct a new dataset, REAL-PS4, by curating four public datasets (AISHELL-4, AliMeeting, AMI, CHiME-6) to create overlapping speech mixtures with ground-truth transcripts and VAD labels. Experiments are conducted on the REAL-T benchmark. The results show consistent improvements over the official REAL-T baselines (BSRNN_TFMAP and BSRNN_EMB) across all four evaluation metrics (TER, SIM, DNSMOS OVRL, Timing F1). Notably, the system achieved 2nd place on the REAL-T challenge leaderboard, with the best speaker similarity and timing F1 scores among all submitted systems. The ablation and per-dataset analysis provide reasonable insights into where the model succeeds (cleaner recordings like AMI) and struggles (noisy/far-field like AISHELL-4).
Reproducibility
The paper provides a link to the code repository (GitHub) and the dataset (HuggingFace). The methodology is described in sufficient detail, including the specific loss formulations and the use of frozen pre-trained models (Whisper, ECAPA-TDNN). The training setup (fine-tuning only the separator) is clearly defined. This ensures a high degree of reproducibility for other researchers.
Limitations
The primary limitation is the reliance on the quality of the source datasets for the REAL-PS4 corpus. If the original diarization or transcript annotations contain errors, these propagate into the training data. Additionally, the use of black-box proxies (Whisper, DNSMOS) introduces potential misalignment; for instance, Whisper might hallucinate content or fail on heavy noise, and DNSMOS is a heuristic metric that may not perfectly correlate with human perception of extraction quality versus just audio fidelity. The paper does not include a detailed ablation study showing the marginal gain of each specific loss component, making it hard to isolate the contribution of the perceptual or VAD losses versus the linguistic/speaker losses.
Broader impact
This work addresses a critical bottleneck in speech processing: the lack of large-scale, real-world training data for TSE. By demonstrating that proxy supervision can effectively bridge the gap between simulated and real domains, it enables more robust deployment of TSE systems in real-world applications like meeting transcription and hearing aids. The release of the REAL-PS4 dataset also benefits the community by providing a standardized, large-scale resource for training and evaluating TSE models in realistic conditions. This paper presents a practical and effective proxy-supervised training framework for target speaker extraction, successfully leveraging multiple auxiliary signals to achieve state-of-the-art performance on real-world conversational speech benchmarks.
Author of this paper? Send a correction.