SLM-SS: Token-Based Speech Separation with Speech Language Models
Abstract
Speech separation (SS) has witnessed substantial progress in recent years with the advent of neural network-based methods, achieving notable improvements on signal-level evaluation metrics. Existing speech separation systems are mostly optimized with signal-level objectives, and often yield outputs that are acceptable in waveform metrics but suboptimal for intelligibility-sensitive downstream tasks. To address this gap, we propose SLM-SS, a token-based generative speech separation framework that models multi-talker mixtures with speech language models. We first discretize clean speech into multi-codebook token sequences with neural codecs and serialize multi-speaker targets via Serialized Output Training (SOT). An encoder-decoder autoregressive speech language model predicts the zero-order sequence to establish speaker ordering and coarse linguistic content, and a non-autoregressive module then refines higher-order residual codebooks for efficient reconstruction. We further introduce a per-talker non-autoregressive strategy that simplifies high-order modeling and improves the stability. On Libri2Mix, SLM-SS with per-talker NAR achieves $6.92\%$ WER, approaching codec-only reconstruction. The WavTokenizer variant eliminates NAR refinement and achieves the highest mean subjective quality score among evaluated separation systems. Its UTMOS and DNSMOS advantages over discriminative baselines persist on unseen noisy and Chinese mixtures without fine-tuning.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces SLM-SS, a token-based generative speech separation framework that utilizes speech language models to predict neural codec tokens, achieving competitive intelligibility and subjective quality scores while offering a flexible architecture for variable-talker scenarios, though it faces challenges in computational efficiency and signal fidelity due to codec discretization limits.
Methodology
The paper proposes SLM-SS, a generative speech separation framework that leverages Speech Language Models (SLMs) to predict discrete neural codec tokens instead of continuous waveforms. The core method involves discretizing clean speech into multi-codebook sequences using Encodec or WavTokenizer, serializing multi-speaker targets via Serialized Output Training (SOT), and using an autoregressive (AR) encoder-decoder to predict the zero-order codebook. A non-autoregressive (NAR) module then refines higher-order residual codebooks. The introduction of a "per-talker NAR" strategy is a key methodological contribution, simplifying high-order modeling by decoupling speaker transitions from residual refinement. The approach effectively bridges the gap between signal-level separation and intelligibility-oriented downstream tasks by operating in the discrete token space.
Results & evaluation
Experiments are conducted on the LibriMix dataset (2-mix and 3-mix). The paper provides a comprehensive comparison against strong discriminative baselines (BSRNN, SepFormer, TF-LocoFormer, MossFormer2). Results show that SLM-SS achieves competitive WER and subjective quality scores, with the WavTokenizer variant achieving the highest mean subjective quality score. The inclusion of subjective listening tests with 30 volunteers and statistical significance testing (Wilcoxon signed-rank) adds rigor. Ablations on temperature, number of codebooks, and model parameters are provided. The evaluation of variable talker counts (up to 4) demonstrates the framework's flexibility.
Reproducibility
The paper provides detailed architectural descriptions (WavLM encoder, Whisper-style decoder), training hyperparameters (learning rate, beam size, epochs), and dataset preprocessing steps (VAD segmentation). However, specific code availability is not explicitly confirmed in the text, and some implementation details regarding the exact initialization of the NAR module and the specific "staged training" strategy are described at a high level. The reliance on specific pre-trained models (WavLM, Whisper, Encodec, WavTokenizer) aids reproducibility.
Limitations
The primary limitation is the computational cost of autoregressive decoding, which limits real-time applicability compared to discriminative methods. The method is constrained by the quality of the neural codec; truncating Encodec to 8 codebooks introduces irreversible distortion that caps the upper bound of signal fidelity (PESQ). The paper acknowledges that WavTokenizer, while better for subjective quality, has higher WER, indicating a trade-off between perceptual quality and linguistic accuracy. The experiments are limited to the LibriMix dataset, which consists of clean speech mixtures; performance on noisy or reverberant environments is not extensively tested in the main results (though mentioned as a future direction or minor test).
Broader impact
This work contributes to the trend of using discrete token representations for speech processing, aligning with the broader shift towards LLM-based audio models. It offers a viable alternative to traditional discriminative separation for applications where intelligibility is paramount, such as multi-talker ASR. The framework's ability to handle variable numbers of talkers without explicit speaker count estimation is a significant practical advantage. The paper introduces SLM-SS, a token-based generative speech separation framework that utilizes speech language models to predict neural codec tokens, achieving competitive intelligibility and subjective quality scores while offering a flexible architecture for variable-talker scenarios, though it faces challenges in computational efficiency and signal fidelity due to codec discretization limits.
Author of this paper? Send a correction.