GSLM2: Multi-Stream Speech Representation Learning and Generative Spoken Language Modeling
Abstract
Generative spoken language models (SLMs) offer a route to learning linguistic structure purely from speech, yet they lag far behind text-based models in both capability and data scaling efficiency. We identify two bottlenecks underlying this gap: the limited linguistic quality of existing speech units and the reliance on a single stream of discrete units to represent speech. Multiple codebooks are widely used to encode acoustic features but rarely to represent speech semantics. We present GSLM2, which learns and models complementary properties of speech at multiple levels of abstraction. Its encoder, SpidR 2.0, uses self-supervised learning with differentiable prediction heads to induce high-quality discrete units at multiple levels. A multi-stream SLM then jointly models these units using a shared temporal backbone, stream-specific predictors, and an autoregressive head that captures cross-stream dependencies. Across training data sizes and model scales, up to 6M hours and 6.5B parameters, GSLM2 consistently improves linguistic competence while scaling more efficiently with data. As a result, it substantially narrows the gap between spoken and text-based LMs; across read and conversational speech domains, we shrink the gap by about two orders of magnitude on sWuggy, sBLIMP, and tSC, while also outperforming competitive speech-text systems on all three tasks. We open-source the training code and model checkpoints at https://github.com/facebookresearch/gslm2.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
GSLM2 introduces a multi-stream speech representation learning framework that significantly improves the data efficiency and linguistic competence of generative spoken language models. By jointly modeling discrete units from multiple encoder layers using a shared backbone and cross-stream generative heads, the method effectively captures complementary phonetic and semantic information, substantially narrowing the performance gap with text-based language models and outperforming existing speech-text systems on key linguistic benchmarks.
Methodology
The paper proposes GSLM2, a framework addressing the data efficiency gap between spoken and text language models. The core methodological contribution is twofold: (1) SpidR 2.0, an encoder pretraining method that uses differentiable prediction heads to induce high-quality discrete units at multiple levels of abstraction (layers) simultaneously, replacing post-hoc k-means with joint optimization; and (2) a multi-stream SLM architecture that jointly models these unit streams using a shared temporal backbone, stream-specific predictors, and an autoregressive generative head to capture cross-stream dependencies. The use of entropy regularization to stabilize code distributions and the specific training objective combining single-stream, self-distillation, and multi-stream generative losses are technically sound and well-motivated.
Results & evaluation
The experiments are extensive, scaling up to 6M hours of data and 6.5B parameters. The paper demonstrates that GSLM2 consistently improves linguistic competence and scales more efficiently with data compared to single-stream baselines. It claims to shrink the gap with text-based LMs by about two orders of magnitude on key benchmarks (sWuggy, sBLIMP, tSC) and outperforms competitive speech-text systems. The comparison against both speech-only and speech-text baselines provides a strong evaluation of the textless approach.
Reproducibility
The authors state that they open-source the training code and model checkpoints at the provided GitHub URL. The paper provides detailed mathematical formulations for the encoder pretraining and SLM training objectives, which aids in understanding the implementation. However, specific hyperparameters for the 6M hour runs may be complex to replicate without the full codebase, but the release of checkpoints is a significant positive factor.
Limitations
The paper focuses primarily on English speech (Libri-Light/LibriSpeech). While it claims generalizability, the evaluation is limited to specific linguistic tasks (sWuggy, sBLIMP, tSC) which may not fully capture the breadth of spoken language understanding. The computational cost of training multi-stream models at scale is high, though the data efficiency gains are highlighted. The paper is an arXiv preprint, so peer review status is pending.
Broader impact
This work is significant for the field of speech AI as it provides a scalable, textless approach to learning linguistic structure. By narrowing the gap with text LMs, it enables more robust spoken dialogue systems that do not rely on transcriptions, which is crucial for low-resource languages or real-time applications. The open-sourcing of the code and checkpoints will likely accelerate research in multi-stream speech representation learning. GSLM2 introduces a multi-stream speech representation learning framework that significantly improves the data efficiency and linguistic competence of generative spoken language models. By jointly modeling discrete units from multiple encoder layers using a shared backbone and cross-stream generative heads, the method effectively captures complementary phonetic and semantic information, substantially narrowing the performance gap with text-based language models and outperforming existing speech-text systems on key linguistic benchmarks.
Author of this paper? Send a correction.