T5Gemma-TTS Technical Report
Abstract
Autoregressive neural codec language models have shown strong zero-shot voice cloning ability, but decoder-only architectures treat input text as a prefix that competes with the growing audio sequence for positional capacity, weakening text conditioning over long utterances. We present T5Gemma-TTS, an encoder-decoder codec language model that maintains persistent text conditioning by routing bidirectional text representations through cross-attention at every decoder layer. Built on the T5Gemma pretrained encoder-decoder backbone (2B encoder + 2B decoder; 4B parameters), it inherits rich linguistic knowledge without phoneme conversion and processes text directly at the subword level. To improve duration control, we introduce Progress-Monitoring Rotary Position Embedding (PM-RoPE) in all 26 cross-attention layers, injecting normalized progress signals that help the decoder track target speech length. Trained on 170,000 hours of multilingual speech in English, Chinese, and Japanese, T5Gemma-TTS achieves a statistically significant speaker-similarity gain on Japanese over XTTSv2 (0.677 vs. 0.622; non-overlapping 95% confidence intervals) and the highest numerical Korean speaker similarity (0.747) despite Korean not being included in training, although this margin over XTTSv2 (0.741) is not statistically conclusive. It also attains the lowest numerical Japanese character error rate among five baselines (0.126), though this ranking should be interpreted cautiously because of partial confidence-interval overlap with Kokoro. English results on LibriSpeech should be viewed as an upper-bound estimate because LibriHeavy is a superset of LibriSpeech. Using the same checkpoint, disabling PM-RoPE at inference causes near-complete synthesis failure: CER degrades from 0.129 to 0.982 and duration accuracy drops from 79% to 46%. Code and weights are available at https://github.com/Aratako/T5Gemma-TTS.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this work is the development of T5Gemma-TTS, a novel encoder-decoder model that enhances multilingual zero-shot text-to-speech synthesis through innovative architectural improvements and rigorous experimental validation. This research represents a meaningful advancement in the field of speech synthesis, addressing key challenges and setting a foundation for future exploration in multilingual and cross-lingual applications.
Methodology
The paper introduces T5Gemma-TTS, an encoder-decoder model that effectively addresses the limitations of autoregressive decoder-only architectures by maintaining persistent text conditioning through cross-attention mechanisms. The integration of Progress-Monitoring Rotary Position Embedding (PM-RoPE) is a significant methodological advancement, allowing for improved duration control during speech synthesis. The model's architecture is well-founded on the T5Gemma pretrained backbone, which enhances its linguistic capabilities without requiring phoneme conversion. The methodology is robust and clearly articulated, demonstrating a thoughtful approach to overcoming existing challenges in zero-shot TTS.
Results & evaluation
The experimental evaluation is comprehensive, involving a substantial training dataset of 170,000 hours of multilingual speech. The results indicate statistically significant improvements in speaker similarity and character error rates compared to existing models. The paper provides detailed comparisons against multiple baselines, showcasing the model's effectiveness across different languages, including Japanese, Chinese, and Korean. The use of confidence intervals adds rigor to the statistical claims, although some results should be interpreted cautiously due to overlapping intervals.
Reproducibility
The authors have made the model weights and code publicly available, which is a positive step towards reproducibility. However, the paper would benefit from more detailed implementation specifics and hyperparameter settings to facilitate easier replication of results by other researchers.
Limitations
The paper acknowledges several limitations, including higher word error rates on unseen European languages and a real-time factor that may not meet the demands of real-time applications. Additionally, the authors note that the model's performance on certain metrics may be influenced by the codec's limitations, indicating areas for future improvement.
Broader impact
The potential for misuse of zero-shot voice cloning technology is a significant concern, as highlighted by the authors. They emphasize the need for ethical considerations and safeguards in deploying such technologies, which is crucial given the implications for privacy and security. The authors advocate for responsible use and further research into detection methods for synthetic speech. The main contribution of this work is the development of T5Gemma-TTS, a novel encoder-decoder model that enhances multilingual zero-shot text-to-speech synthesis through innovative architectural improvements and rigorous experimental validation. This research represents a meaningful advancement in the field of speech synthesis, addressing key challenges and setting a foundation for future exploration in multilingual and cross-lingual applications.
Author of this paper? Send a correction.