Position-Aware Target Speaker Extraction for Long-Form Multi-Party Conversations: A Diarization-Free Framework for ASR
Abstract
In long-form multi-party conversations, highly imbalanced speaker activity and frequent overlap make it difficult to identify "who spoke when and what". Sliding-window continuous speech separation (CSS) mitigates sparse supervision, but often suffers from cross-window speaker inconsistency and residual crosstalk, which in practice requires diarization for reliable speaker attribution. Motivated by the stability of speakers' directions of arrival (DOAs) in meetings, we propose PATSE, a multi-channel Position-Aware Target Speaker Extraction front-end that uses DOA as a spatial prior to directly extract the speech of each target speaker. PATSE combines a DOA-guided spatial encoder and conditioner to generate speaker-attributed streams, from which speaker activity can be inferred via simple post-processing (e.g., VAD) without explicit diarization. Experiments on both replayed and real conversations show consistent ASR gains outperforming CSS and diarization-based pipelines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a practical and effective framework for diarization-free target speaker extraction using DOA priors, demonstrating significant ASR gains in multi-party conversations through the novel integration of spatial conditioning into continuous speech separation.
Methodology
The paper proposes PATSE, a Position-Aware Target Speaker Extraction framework that leverages Direction of Arrival (DOA) as a spatial prior to condition a separation backbone (TIGER). The core methodological contribution is the integration of a DOA-guided spatial encoder and conditioner (using FiLM modulation) into a continuous speech separation pipeline. This allows the model to extract specific speaker streams directly, bypassing the need for explicit speaker diarization. The approach is technically sound, combining established multi-channel features (IPD, TPD) with modern deep separation architectures. However, the novelty is moderate as DOA-conditioned extraction is a known paradigm in the speech processing community; the primary innovation lies in its specific application to long-form, diarization-free ASR pipelines and the integration with the TIGER backbone.
Results & evaluation
The experimental evaluation is robust and addresses a significant gap in the field: the lack of real-world datasets with ground-truth DOA labels. The authors introduce LibriReplay-DOA, a replayed dataset, and evaluate on TEIDAN, a real-world conversational dataset. Results demonstrate consistent Word Error Rate (WER) improvements over strong baselines including CSS (TIGER), Sortformer+GSS, and FastMNMF. The comparison against CSS with oracle speaker assignment is particularly compelling, highlighting the inherent instability of sliding-window separation without spatial priors. The evaluation covers various angular configurations and overlap ratios, providing a comprehensive view of performance under different acoustic conditions.
Reproducibility
The paper provides detailed architectural descriptions, including the specific implementation of the spatial encoder, conditioner, and loss functions. The authors release the LibriReplay-DOA dataset and a demo page, which significantly aids reproducibility. The use of standard components like TIGER and Silero-VAD also supports reproducibility. However, the exact hyperparameters for the training of the PATSE module on top of TIGER (e.g., learning rate schedules, specific optimizer settings beyond the initial LR) could be more detailed.
Limitations
The method relies on the availability of accurate DOA information. While DOAs are stable in meeting scenarios, they may vary in more dynamic environments. The performance on LibriReplay-DOA, while strong, is based on replayed audio, which does not fully capture the complex reverberation and noise characteristics of real spontaneous conversations, although TEIDAN results mitigate this concern. The approach assumes speakers are stationary or move slowly enough for DOA estimation to remain valid during the extraction window.
Broader impact
This work has significant implications for automatic speech recognition in multi-party settings, such as meeting transcription systems. By eliminating the need for explicit diarization, it simplifies the pipeline and improves robustness to diarization errors. The release of LibriReplay-DOA provides a valuable resource for the community to benchmark DOA-based methods on real-room recordings, fostering further research in spatial audio processing. This paper presents a practical and effective framework for diarization-free target speaker extraction using DOA priors, demonstrating significant ASR gains in multi-party conversations through the novel integration of spatial conditioning into continuous speech separation.
Author of this paper? Send a correction.