SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation
Abstract
Self-supervised learning (SSL) is standard for speech representation learning, but mainstream models are designed around single-speaker audio, limiting their usefulness in multi-speakers scenarios. We present SepRQ, an open-source SSL framework that replaces masked prediction with a pseudo-source-separation objective over frozen random-projection codebooks. By adopting a novel mask-free, multiresolution approach, SepRQ achieves state-of-the-art performance in Speaker Diarization and Speech Separation on the SUPERB benchmark, surpassing WavLM and other cocktail-party derived SSLs at both Base and Large scales, while requiring only 85.68M inference parameters. SepRQ also demonstrates strong performance across target-speaker tasks requiring enrollment (such as Target-Speaker Automatic Speech Recognition), and on the challenging multi-domain DIHARD 3 diarization dataset. Notably, we report strong separation capabilities on three-speaker mixtures (WSJ0-3Mix), where current SSL literature struggles. While cocktail-party SSLs remain scarce and closed-source, limited to C-HuBERT and the enrollment-based SA-WavLM, we open-source SepRQ to the community.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SepRQ introduces a novel mask-free, multiresolution pseudo-source-separation objective for self-supervised speech representation learning, achieving state-of-the-art performance in multi-speaker tasks like diarization and separation while maintaining high efficiency and open-source accessibility. The paper addresses a critical gap in SSL for multi-speaker scenarios by moving beyond single-speaker masked prediction to a separation-focused objective. The multiresolution design is a key technical contribution, allowing the model to capture both fine-grained acoustic details and longer-term speaker structure. The rigorous ablation studies and extensive benchmarking, including challenging out-of-domain and three-speaker tasks, provide strong evidence for the effectiveness of the approach. The open-source nature of the work is a significant contribution to the community, fostering further research and development in this area.
Methodology
The paper introduces SepRQ, a self-supervised learning (SSL) framework for multi-speaker speech that replaces standard masked prediction with a pseudo-source-separation (PSS) objective. The core innovation is the application of this separation objective across multiple temporal resolutions (multiresolution) by progressively downsampling the encoder's hidden states. The model uses frozen Random Vector Quantizers (RVQs) to create discrete targets for each speaker stream, avoiding the need for offline clustering (as in HuBERT) or enrollment embeddings (as in SA-WavLM). The "mask-free" aspect is a significant methodological choice, arguing that masking suppresses necessary content for disentanglement in mixtures. The use of Permutation Invariant Training (PIT) at each resolution level to handle speaker identity ambiguity is a robust design choice. The architecture is based on a 12-layer Conformer, modified to operate at 50 Hz to balance fine-grained acoustic detail with computational efficiency.
Results & evaluation
The experimental evaluation is comprehensive and rigorous. The authors conduct a systematic ablation study to isolate the contributions of the PSS objective, the mask-free approach, and the multiresolution strategy. They demonstrate that the gains are not merely due to domain matching (training on mixtures) but specifically due to the separation objective. The model is evaluated on the SUPERB and TS-SUPERB benchmarks, showing state-of-the-art performance in Speaker Diarization (SD), Speech Separation (SS), and Target-Speaker ASR (TS-ASR) compared to WavLM, C-HuBERT, and SA-WavLM. Crucially, the paper includes out-of-domain (OOD) evaluations on DIHARD 3 and WSJ0-3Mix, where SepRQ shows particularly strong performance on three-speaker mixtures, a scenario where other SSL models struggle. The efficiency of the model (85.68M inference parameters) is also highlighted as a practical advantage.
Reproducibility
The paper is highly reproducible. The authors explicitly state that SepRQ is open-source and provide a link to the project page (https://sevkod.github.io/SepRQ/). Detailed hyperparameters, training schedules, and architectural modifications are provided. The use of standard datasets (LibriSpeech, WHAM!, DIHARD 3) and established benchmarks (SUPERB, TS-SUPERB) further enhances reproducibility. The disclosure of using LLMs for editing is transparent, though it does not affect the technical reproducibility of the model itself.
Limitations
The primary limitation acknowledged by the authors is the performance gap on single-speaker tasks, such as standard ASR, where SepRQ underperforms compared to models like WavLM. This suggests that the multi-speaker focus may come at the cost of general single-speaker representation quality. Additionally, the model relies on synthetic mixture generation during pre-training, which may not fully capture the complexity of real-world conversational dynamics, although OOD results are promising. The "mask-free" approach, while effective for separation, might limit the model's robustness to certain types of noise or corruption that masking is typically designed to handle.
Broader impact
This work has significant potential impact on the field of speech processing, particularly for applications involving overlapping speech such as meeting transcription, call center analytics, and assistive listening devices. By providing an open-source, efficient, and state-of-the-art SSL model for multi-speaker scenarios, it lowers the barrier to entry for researchers and developers working on cocktail-party problems. The multiresolution approach offers a new perspective on how to capture temporal dependencies in speech, which could inspire further research in SSL architectures. The strong performance on three-speaker mixtures is particularly valuable, as most existing SSL models are limited to two-speaker scenarios. SepRQ introduces a novel mask-free, multiresolution pseudo-source-separation objective for self-supervised speech representation learning, achieving state-of-the-art performance in multi-speaker tasks like diarization and separation while maintaining high efficiency and open-source accessibility. The paper addresses a critical gap in SSL for multi-speaker scenarios by moving beyond single-speaker masked prediction to a separation-focused objective. The multiresolution design is a key technical contribution, allowing the model to capture both fine-grained acoustic details and longer-term speaker structure. The rigorous ablation studies and extensive benchmarking, including challenging out-of-domain and three-speaker tasks, provide strong evidence for the effectiveness of the approach. The open-source nature of the work is a significant contribution to the community, fostering further research and development in this area.
Author of this paper? Send a correction.