PromptSep-Speaker: Generative Target Speaker Extraction via Multimodal Prompting and Semantic Alignment
Abstract
Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
PromptSep-Speaker introduces a generative latent-diffusion framework for target speaker extraction that unifies four interchangeable cues (speaker ID, transcript, enrollment, and caption) and employs a novel auxiliary alignment loss to supervise cross-attention between text cues and WavLM features, achieving state-of-the-art performance in both objective metrics and subjective listening tests across diverse acoustic conditions.
Methodology
The paper proposes PromptSep-Speaker, a generative target speaker extraction (TSE) model based on latent diffusion. The core methodological contribution is the unification of four distinct cue modalities (speaker temporal order, transcript, enrollment audio, and speaker caption) within a single framework. A significant technical innovation is the "alignment module," which uses a frozen WavLM representation of the mixture to cross-attend to text-based cues (transcript/speaker ID). This is coupled with a novel auxiliary alignment loss that supervises the cross-attention maps to ensure the model correctly localizes the target speaker in the time domain without requiring extra labeling. The use of a DiT (Diffusion Transformer) backbone fine-tuned from a pre-trained source separation model is a strong architectural choice, leveraging prior knowledge of audio generation. The integration of semantic features via WavLM to guide attention is a clever way to bridge the gap between high-level linguistic cues and low-level acoustic separation.
Results & evaluation
The evaluation is rigorous and comprehensive. The authors test on four distinct datasets (CapSpeech-Mix, Libri2Mix, LibriTTS-R-Mix, and an internal set), covering both in-domain and out-of-domain scenarios. They compare against a diverse set of baselines, including mask-based TSE systems (DAE-TSE, SoloSpeech, WeSep) and generative universal source separators (PromptSep, FlowSep, SAM-Audio). The results show consistent state-of-the-art performance across SI-SDR, mel-L2, DNSMOS, and F1 decision error metrics. The inclusion of a subjective listening test with 380 subjects and 4,515 submissions adds significant weight to the objective metrics, confirming that the generative approach yields better perceptual quality and speaker accuracy than mask-based alternatives. The ablation studies effectively demonstrate the necessity of the auxiliary alignment loss and the specific direction of cross-attention.
Reproducibility
The paper provides a good level of detail regarding the architecture (DiT, DAC-VAE, WavLM) and training hyperparameters (learning rate, sampler steps). However, the reliance on a "large internal collection of licensed data" for training limits full reproducibility by external groups. The specific composition of the 4.8M speech samples and the simulation pipeline details are high-level. While the demo page is provided, the absence of a public code repository or pre-trained weights for the specific PromptSep-Speaker model (distinct from the base models) is a minor drawback, though common for industry-affiliated papers at the arXiv stage.
Limitations
The model is computationally expensive due to the diffusion process, which may limit real-time application compared to mask-based models. The evaluation is primarily on English speech; generalization to other languages is not tested. The "internal set" is not publicly available, making it impossible for third parties to verify those specific results. The paper does not extensively discuss the failure modes of the generative approach, such as hallucinations or artifacts in complex acoustic environments, beyond general statements about quality.
Broader impact
This work has significant implications for the field of speech processing by demonstrating that generative models can outperform traditional discriminative (mask-based) approaches in target speaker extraction. The multimodal prompting capability makes the system more versatile and user-friendly, as users can provide whichever cue is most available (e.g., a transcript from a meeting or a voice sample). This could lead to more robust meeting transcription systems, assistive listening devices, and content creation tools. The alignment loss technique could be adapted to other multimodal generation tasks where precise temporal grounding of text to audio is required. PromptSep-Speaker introduces a generative latent-diffusion framework for target speaker extraction that unifies four interchangeable cues (speaker ID, transcript, enrollment, and caption) and employs a novel auxiliary alignment loss to supervise cross-attention between text cues and WavLM features, achieving state-of-the-art performance in both objective metrics and subjective listening tests across diverse acoustic conditions.
Author of this paper? Send a correction.