Audio ML PapersLandmark papers
← All papers
RESEARCH PAPER / 2026-11-13

Factorized Self-Supervised Speech Tokenization

Benjamin Van Niekerk, Jean-Philippe Letendre, Nicol Visser, Hugo Seute, Herman Kamper, Mirco Ravanelli

Ubisoft La Forge +3 · NeurIPS 2026

Affiliations: Ubisoft La Forge, Concordia University, Mila, Stellenbosch University

Abstract

Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker's voice and delivery. We propose Facto: a factorized tokenizer that disentangles time-varying information (content and prosody) from static components (speaker identity and recording conditions). We build on a recent linear factorization method for self-supervised features, extending it to handle unseen speakers. Then, we cluster the factorized space and train a lightweight decoder to reconstruct audio from the resulting tokens. By removing the static components before clustering, our tokens are more robust to small noise perturbations. We validate this by testing token stability under various noise conditions. Next, we theoretically analyze Facto's convergence and scaling properties, quantifying how the factorization improves with the number of training speakers. Finally, we evaluate Facto on a range of downstream tasks. We show that greater token stability translates into improved text-to-speech performance and competitive results in voice conversion and spoken language modeling. Code and checkpoints at https://github.com/ubisoft/ubisoft-laforge-facto/ and demo page at https://ubisoft-laforge.github.io/speech/facto/.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

Facto proposes a factorized speech tokenizer that disentangles content from speaker identity using linear factorization of self-supervised features, providing theoretical guarantees on convergence and stability. The paper makes a significant contribution by extending linear factorization to unseen speakers and rigorously analyzing the scaling properties with the number of training speakers, resulting in more robust tokens for downstream tasks like TTS and voice conversion.

Methodology

The paper proposes "Facto," a factorized speech tokenizer that disentangles time-varying content/prosody from static speaker/channel information. The core method builds on a linear factorization of self-supervised features (WavLM), extending it to handle unseen speakers via a linear content estimator. The methodology is theoretically grounded, providing convergence analysis for the factorization and a bias-variance trade-off analysis for the content estimator. The approach uses PCA and k-means clustering on the factorized content space, followed by a lightweight HiFiGAN-based decoder with FiLM conditioning for speaker identity reintroduction. The theoretical contributions (Theorem 1 and 2, Corollary 1) are a strong differentiator, offering rigorous bounds on error scaling with the number of training speakers.

Results & evaluation

The experiments validate the theoretical predictions, showing that the factorization stabilizes as the number of speakers increases (O(1/K) decay). The paper demonstrates improved token stability under noise perturbations compared to baselines. Downstream tasks include text-to-speech, voice conversion, and spoken language modeling. The results show competitive performance, with clear improvements in TTS due to token stability. The use of LibriSpeech and standard metrics provides a solid empirical foundation.

Reproducibility

The paper provides code and checkpoints on GitHub and a demo page. The methodology is described in sufficient detail, including hyperparameters (e.g., rank R=12, WavLM layer 6) and training setups. The theoretical derivations are provided in the main text or appendices, enhancing reproducibility.

Limitations

The method relies on the quality of the self-supervised features (WavLM) and the linear factorization assumption, which may not capture all non-linear interactions. The content estimator is linear, which might limit its ability to handle highly complex speaker variations. The paper focuses on English speech (LibriSpeech), and generalization to other languages or noisy real-world environments beyond small perturbations is not extensively tested.

Broader impact

The work contributes to the field of speech tokenization by providing a principled, theoretically sound approach to disentangling content and speaker identity. This can benefit downstream tasks like TTS, voice conversion, and speech language modeling by providing more robust and compact representations. The emphasis on stability and theoretical guarantees sets a high standard for future work in speech representation learning. Facto proposes a factorized speech tokenizer that disentangles content from speaker identity using linear factorization of self-supervised features, providing theoretical guarantees on convergence and stability. The paper makes a significant contribution by extending linear factorization to unseen speakers and rigorously analyzing the scaling properties with the number of training speakers, resulting in more robust tokens for downstream tasks like TTS and voice conversion.

Author of this paper? Send a correction.