ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure
Abstract
Neural speech codecs face a fundamental tension in the language-model era: tokens that support high-fidelity reconstruction are not necessarily easy for autoregressive models to predict. Our controlled analysis of diverse codec and self-supervised speech representations shows that clearer phoneme structure before discrete code assignment is consistently associated with easier autoregressive token prediction. Yet phoneme structure alone is insufficient for high-fidelity reconstruction, which also requires reconstruction-relevant acoustic detail. Guided by this observation, we introduce ReLMCodec, a low-bitrate single-codebook speech codec built upon a preserve--control--refine principle: it preserves the linguistic organization of frozen self-supervised learning (SSL) features at the quantizer input, controls reconstruction-driven drift through Pre-quantization Anchor-Preserving Adaptation (PAPA), and refines the quantized latent space with a training-only WavLM-Large L24 teacher to reduce phoneme-level token fragmentation. Together, these components allow acoustic detail to support waveform reconstruction while keeping the resulting token sequence predictable for autoregressive models. At 650 and 800 bps, ReLMCodec moves the empirical single-stream predictability--reconstruction frontier in our evaluations, with gains that carry over to downstream text-to-speech (TTS) synthesis in both intelligibility and speaker similarity.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ReLMCodec introduces a novel single-codebook speech codec that leverages a frozen SSL anchor and residual adaptation to preserve phoneme structure, significantly improving the predictability of speech tokens for autoregressive models while maintaining high reconstruction fidelity. The paper makes a significant technical contribution by identifying and addressing the tension between reconstruction quality and token predictability in neural speech codecs. The diagnostic study providing empirical evidence linking pre-quantization phoneme structure to token predictability is a valuable insight for the community. The proposed ReLMCodec architecture, particularly the PAPA mechanism, offers a practical and effective solution to this problem. The comprehensive evaluation, including both reconstruction metrics and downstream TTS performance, strongly supports the claims. The work is well-written, clearly motivated, and technically rigorous. It represents a solid advancement in the field of speech tokenization for LLM-based generation.
Methodology
The paper proposes ReLMCodec, a single-codebook speech codec designed to optimize the trade-off between waveform reconstruction quality and autoregressive token predictability. The core methodological contribution is the "preserve--control--refine" principle. It preserves linguistic structure by anchoring the quantizer input to frozen W2v-BERT 2.0 features. It controls reconstruction drift via Pre-quantization Anchor-Preserving Adaptation (PAPA), which adds a scaled residual correction to the anchor rather than learning the full representation. It refines the latent space using a training-only WavLM-Large teacher to reduce phoneme-level fragmentation. The approach is technically sound and addresses a specific, well-defined problem in the intersection of speech coding and LLM-based generation. The use of a frozen SSL anchor to maintain phoneme structure while allowing acoustic detail to be learned via a residual path is a clever and effective architectural choice.
Results & evaluation
The experimental evaluation is comprehensive. The authors conduct a controlled diagnostic study across 24 different codec and SSL representations, establishing a strong correlation between pre-quantization phoneme structure (measured by KNN accuracy) and token predictability (P-ACC/P-PPL). This diagnostic provides a strong theoretical grounding for the design choices. The end-to-end results on LibriSpeech show that ReLMCodec achieves state-of-the-art performance at low bitrates (650/800 bps) in terms of WER, PESQ, and speaker similarity compared to baselines like X-Codec2, FocalCodec, and SemantiCodec. The downstream TTS evaluation further validates the utility of the tokens, showing improved intelligibility and speaker similarity. The ablation studies effectively isolate the contributions of PAPA and the JMAS refinement loss.
Reproducibility
The paper provides detailed descriptions of the architecture, training objectives, and evaluation protocols. The appendix includes extensive details on the probing protocol, baseline configurations, and statistical methods. The code release statement indicates that checkpoints and scripts will be available upon acceptance. The use of standard datasets (LibriSpeech) and metrics (WER, PESQ, UTMOS) enhances reproducibility. The matched probing protocol is particularly well-documented, allowing other researchers to replicate the diagnostic analysis.
Limitations
The study is limited to English speech (LibriSpeech). The analysis of phoneme structure relies on forced alignment (MFA), which may not be robust to noisy or out-of-domain data. The teacher selection is limited to W2v-BERT and WavLM; the generalizability to other SSL models is not explored. The paper acknowledges that multilingual and noisy settings are future work. Additionally, the reliance on a training-only teacher adds complexity to the training pipeline, although it does not affect inference cost.
Broader impact
ReLMCodec contributes to the development of efficient and predictable speech tokenizers, which are critical for the advancement of end-to-end speech generation models based on LLMs. By improving the predictability of tokens, it may facilitate the training of smaller, more efficient speech LLMs. The insights gained from the diagnostic study could inform the design of future speech representations. The work has positive implications for the field of audio AI, promoting more efficient and linguistically aware speech processing. ReLMCodec introduces a novel single-codebook speech codec that leverages a frozen SSL anchor and residual adaptation to preserve phoneme structure, significantly improving the predictability of speech tokens for autoregressive models while maintaining high reconstruction fidelity. The paper makes a significant technical contribution by identifying and addressing the tension between reconstruction quality and token predictability in neural speech codecs. The diagnostic study providing empirical evidence linking pre-quantization phoneme structure to token predictability is a valuable insight for the community. The proposed ReLMCodec architecture, particularly the PAPA mechanism, offers a practical and effective solution to this problem. The comprehensive evaluation, including both reconstruction metrics and downstream TTS performance, strongly supports the claims. The work is well-written, clearly motivated, and technically rigorous. It represents a solid advancement in the field of speech tokenization for LLM-based generation.
Author of this paper? Send a correction.