DisCo-Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec
Abstract
Recent codec-based language models~(LMs) have revolutionized text-to-speech~(TTS). However, since standard codecs tightly couple timbre and prosody, continuation-based LMs inevitably replicate this entanglement, hindering independent control. Recent efforts attempt to break this entanglement via codec design, but insufficient decoupling remains a critical bottleneck. To tackle this challenge, we propose DisCo-Speech, a zero-shot controllable TTS framework that enables prosody control and voice cloning via a disentangled speech codec (DisCodec) and an LM-based generator. The core component, DisCodec, contains two core stages: 1) Tri-factor disentanglement, which explicitly factorizes speech into content, prosody, and timbre subspaces via parallel encoders and hybrid losses; and 2) Fusion and reconstruction, which fuses content and prosody into unified content-prosody tokens suitable for LM prediction, while jointly optimizing reconstruction quality to resolve the disentanglement-reconstruction trade-off. With this design, the LM performs prosodic continuation from a style prompt while the decoder handles target timbre injection, enabling flexible zero-shot control. Experiments show that DisCo-Speech matches state-of-the-art voice cloning performance while outperforming baselines in zero-shot prosody control. By resolving the core entanglement at the codec level, DisCo-Speech provides a robust foundation for controllable speech synthesis. Audio samples are available at https://github.com/disco-speech/DisCo-Speech, and the code and weights will be released at the same link.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DisCo-Speech, a novel framework for zero-shot controllable speech generation that achieves independent control over speaker timbre and speaking prosody through a disentangled speech codec. This work represents a significant step forward in the field of text-to-speech synthesis, addressing critical challenges in disentanglement and control, and providing a robust foundation for future research and applications.
Methodology
The proposed methodology of DisCo-Speech is innovative, focusing on disentangling speech attributes into content, prosody, and timbre through a two-stage training paradigm. The tri-factor disentanglement approach is a significant advancement over existing methods, allowing for independent control over speech generation. The use of hybrid losses and parallel encoders is well-justified, addressing the disentanglement-reconstruction trade-off effectively. The integration of a standard LM for prosodic continuation and a specialized decoder for waveform synthesis is a thoughtful design choice that enhances the flexibility of the system.
Results & evaluation
The experimental evaluation is thorough, utilizing a diverse dataset and comparing DisCo-Speech against state-of-the-art models. The results demonstrate competitive performance in voice cloning and prosody control, with clear metrics provided for reconstruction quality and controllability. The use of both objective and subjective evaluation metrics strengthens the credibility of the findings. However, more extensive comparisons with a broader range of existing methods could provide deeper insights into its relative performance.
Reproducibility
The paper provides sufficient detail regarding the architecture, training procedures, and evaluation metrics, which supports reproducibility. The authors also mention plans to release code and weights, which is essential for enabling other researchers to validate the findings and build upon the work. However, the absence of specific details about the training data and preprocessing steps could hinder full reproducibility.
Limitations
The paper acknowledges limitations, including lower speaker similarity compared to multi-stage systems and potential instability in generating exaggerated prosody. The delicate balance between disentanglement and reconstruction fidelity is also highlighted as an ongoing challenge. These limitations suggest areas for future improvement, particularly in enhancing the expressive range and fidelity of the generated speech.
Broader impact
The advancements presented in DisCo-Speech have significant implications for applications in human-computer interaction, entertainment, and accessibility technologies. The ability to generate speech with controlled prosody and timbre could enhance user experience in virtual assistants, audiobooks, and language learning tools. Furthermore, the framework's potential for zero-shot learning could democratize access to high-quality speech synthesis across diverse languages and dialects. The main contribution of this paper is the introduction of DisCo-Speech, a novel framework for zero-shot controllable speech generation that achieves independent control over speaker timbre and speaking prosody through a disentangled speech codec. This work represents a significant step forward in the field of text-to-speech synthesis, addressing critical challenges in disentanglement and control, and providing a robust foundation for future research and applications.
Author of this paper? Send a correction.