EEND-SAA: Enrollment-Less Main Speaker Voice Activity Detection Using Self-Attention Attractors

Wen-Yung Wu, Pei-Chin Hsieh, Tai-Shih Chi

National Yang Ming Chiao Tung University +3 · arXiv

Affiliations: Institute of Electrical and Computer Engineering, Taiwan under Grant NSTC 113-2221-E-A49-146, This research is supported by the National Science and Technology Council, National Yang Ming Chiao Tung University

Paper

Abstract

Voice activity detection (VAD) is essential in speech-based systems, but traditional methods detect only speech presence without identifying speakers. Target-speaker VAD (TS-VAD) extends this by detecting the speech of a known speaker using a short enrollment utterance, but this assumption fails in open-domain scenarios such as meetings or customer service calls, where the main speaker is unknown. We propose EEND-SAA, an enrollment-less, streaming-compatible framework for main-speaker VAD, which identifies the primary speaker without prior knowledge. Unlike TS-VAD, our method determines the main speaker as the one who talks more steadily and clearly, based on speech continuity and volume. We build our model on EEND using two self-attention attractors in a Transformer and apply causal masking for real-time use. Experiments on multi-speaker LibriSpeech mixtures show that EEND-SAA reduces main-speaker DER from 6.63% to 3.61% and improves F1 from 0.9667 to 0.9818 over the SA-EEND baseline, achieving state-of-the-art performance under conditions involving speaker overlap and noise.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper presents a novel framework for enrollment-less main speaker voice activity detection using self-attention attractors, significantly advancing the state of the art in multi-speaker scenarios. The technical contributions, particularly the dual self-attention mechanism and real-time processing capabilities, position this work as a meaningful addition to the field of audio processing and speech technology.

Methodology

The methodology presented in EEND-SAA is innovative in its approach to voice activity detection by eliminating the need for speaker enrollment, which is a significant limitation in traditional systems. The use of self-attention attractors within a Transformer framework is a novel contribution that enhances the model's ability to distinguish the main speaker from background noise and overlapping speech. The dual self-attention mechanism is particularly noteworthy, as it allows the model to focus on both the main speaker and background speakers simultaneously, improving detection accuracy. The incorporation of causal masking for real-time processing further enhances the practicality of the model in interactive environments. Overall, the methodology is sound and builds effectively on existing work in the field, particularly EEND and its variants.

Results & evaluation

The experimental evaluation is robust, utilizing the LibriSpeech dataset to simulate real-world conditions with overlapping speakers and noise. The authors provide comprehensive results that demonstrate significant improvements in main-speaker detection metrics, such as DER and F1 scores, compared to baseline models. The ablation studies are particularly useful in highlighting the contributions of various components, such as positional encoding and the dual attractor design. However, the paper could benefit from more extensive comparisons with additional state-of-the-art methods to further validate the performance claims.

Reproducibility

The paper provides a detailed description of the model architecture, training procedures, and evaluation metrics, which aids in reproducibility. However, the absence of a publicly accessible code repository or demo limits the ability for other researchers to easily replicate the results. Including such resources would significantly enhance the reproducibility of the findings.

Limitations

One limitation of the proposed approach is its reliance on the quality of the input audio. In extremely noisy environments or with significant overlap from multiple speakers, the model's performance may degrade. Additionally, while the model shows promise in real-time applications, the computational efficiency and latency in practical deployments have not been extensively discussed. The paper also does not address potential biases in the training data that could affect the model's generalizability across different speaker demographics.

Broader impact

The implications of this research are significant for various applications in speech recognition, customer service, and interactive voice systems, where identifying the main speaker in noisy environments is crucial. The enrollment-less approach can facilitate more flexible and user-friendly systems, making them more accessible in real-world scenarios. This work could lead to advancements in smart assistants, meeting transcription tools, and other audio processing applications, ultimately improving user experience and system efficiency. The paper presents a novel framework for enrollment-less main speaker voice activity detection using self-attention attractors, significantly advancing the state of the art in multi-speaker scenarios. The technical contributions, particularly the dual self-attention mechanism and real-time processing capabilities, position this work as a meaningful addition to the field of audio processing and speech technology.

Author of this paper? Send a correction.