FC-TTS: Style and Timbre Control in Zero-Shot Text-to-Speech with Disentangled Speech Representations
Abstract
Recent advances in zero-shot text-to-speech (TTS) have enabled accurate imitation of reference speech in terms of both speaking style and speaker timbre. However, achieving disentangled control over these aspects from separate references remains a challenging task. Several studies have proposed disentangled speech representations that decompose speech into interpretable attributes (e.g., timbre, prosody, and content), providing a promising foundation for TTS with attribute control from separate references. Yet, how to effectively integrate such representations into TTS systems to achieve independent and precise control remains underexplored. In this paper, we present FC-TTS, a zero-shot TTS framework that enables disentangled control of style and timbre by conditioning on two distinct reference utterances. Unlike existing systems that inherit limitations from those pre-trained disentangled representations, FC-TTS introduces key design strategies, including architectural choices, training framework, and auxiliary training objectives, which improve the reliability of attribute separation and dual-reference control. Experiments show that FC-TTS achieves high-fidelity synthesis and competitive zero-shot naturalness, while uniquely supporting consistent and independent manipulation of style and timbre. Audio samples are available at https://qualcomm-ai-research.github.io/fc-tts
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of FC-TTS, a zero-shot TTS framework that allows for independent control over speaking style and timbre, significantly advancing the capabilities of text-to-speech synthesis. The methodology is innovative, addressing critical challenges in the field, and the experimental results demonstrate its effectiveness, marking a notable step forward in TTS technology.
Methodology
The paper introduces FC-TTS, a novel framework for zero-shot text-to-speech synthesis that enables disentangled control over style and timbre using separate reference utterances. The methodology is well-structured, utilizing a two-stage spectrogram generation pipeline, a VQ-VAE-based style encoder, and a conditional consistency loss to enhance the robustness and interpretability of the generated audio. The approach is innovative, addressing the limitations of existing methods that typically entangle style and timbre in a single reference. The proposed architecture and training strategies are clearly articulated and demonstrate a significant advancement in TTS technology.
Results & evaluation
The experimental evaluation is comprehensive, employing a variety of datasets (LibriSpeech and RAVDESS) to assess the performance of FC-TTS against state-of-the-art models. The use of both objective metrics (UTMOS, WER, SPK, MCD) and subjective evaluations (ABX tests) provides a robust framework for measuring the effectiveness of the proposed system. The results indicate that FC-TTS achieves competitive performance while maintaining the ability to manipulate style and timbre independently, which is a critical contribution to the field.
Reproducibility
The paper provides detailed implementation details, including model architecture, training procedures, and hyperparameter settings, which enhance reproducibility. However, the reliance on proprietary datasets and the absence of publicly available checkpoints for some baseline models may limit the ease of reproduction for external researchers.
Limitations
The paper acknowledges limitations such as the dependency on the quality of codec representations, potential issues with generalization to non-English languages, and the ethical implications of using such technology for deepfake generation. These factors highlight the need for further research to address robustness and ethical concerns in TTS applications.
Broader impact
The advancements presented in this paper have significant implications for various applications, including virtual assistants, audiobooks, and accessibility tools. The ability to control style and timbre independently can enhance user experience and personalization in TTS systems. However, the potential for misuse in generating deepfake audio raises ethical concerns that must be addressed in future developments. The main contribution of this paper is the development of FC-TTS, a zero-shot TTS framework that allows for independent control over speaking style and timbre, significantly advancing the capabilities of text-to-speech synthesis. The methodology is innovative, addressing critical challenges in the field, and the experimental results demonstrate its effectiveness, marking a notable step forward in TTS technology.
Author of this paper? Send a correction.