dots.tts Technical Report
Abstract
We present dots.tts, a 2B-parameter continuous autoregressive text-to-speech (TTS) foundation model that models speech in a continuous latent space. Compared with existing continuous autoregressive models, our key innovations are threefold. First, we train an AudioVAE with multiple objectives to build a semantically structured and prediction-friendly continuous speech space. Second, we use full-history conditioning in the flow-matching head to preserve long-range consistency and reduce drift during generation. Third, we apply reward-free self-corrective post-training to the flow-matching head to further improve robustness and acoustic quality. After being trained on a large-scale multilingual corpus, dots.tts achieves the best average performance on Seed-TTS-Eval, with WERs of 0.94%/1.30%/6.60% and SIM scores of 81.0/77.1/79.5 on the zh/en/zh-hard test sets, respectively. Across other benchmarks, dots.tts also consistently demonstrates open-source state-of-the-art performance, exhibiting strong generation stability, voice cloning ability, and emotional expressiveness. For efficient inference, we further apply CFG-aware MeanFlow distillation, enabling low-latency speech generation with first-packet latencies of 85/54 ms in output streaming and dual-streaming modes, respectively. To facilitate reproducible research and practical deployment, we release the training and inference code, together with the pretrained, post-trained, and MeanFlow-distilled checkpoints, under the Apache 2.0 license.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents dots.tts, a state-of-the-art autoregressive TTS system that effectively addresses long-range error accumulation and improves synthesis quality through innovative methodologies. Its comprehensive evaluation and release of resources position it as a significant contribution to the field of machine learning and audio synthesis.
Methodology
The paper introduces a novel autoregressive text-to-speech (TTS) model, dots.tts, which utilizes a continuous latent space to improve upon existing discrete-token TTS systems. The methodology is robust, featuring a multi-objective training approach for the AudioVAE, full-history conditioning for long-range consistency, and a self-corrective post-training stage that enhances the model's robustness and acoustic quality. The integration of a semantic encoder and autoregressive flow-matching head is particularly noteworthy, as it effectively decouples semantic reasoning from acoustic rendering, addressing the long-range error accumulation problem prevalent in continuous autoregressive models.
Results & evaluation
The experiments are extensive, utilizing a large-scale multilingual corpus of 1.5M hours of speech. The results demonstrate state-of-the-art performance across multiple benchmarks, including Seed-TTS-Eval and MiniMax-Speech, with significant improvements in word error rates (WER) and speaker similarity (SIM) scores. The paper effectively compares its results against a variety of existing systems, showcasing the advantages of its approach in terms of stability, expressiveness, and efficiency.
Reproducibility
The authors provide comprehensive details on the training and inference processes, including the release of code and pretrained models under the Apache 2.0 license. This transparency facilitates reproducibility, allowing other researchers to replicate their findings and build upon their work.
Limitations
The paper acknowledges several limitations, including the model's reliance on BPE tokens rather than phonemes, which may lead to data inefficiencies, particularly for under-represented languages. Additionally, the system has not been explicitly tested for style control or instruction-based variations, which could limit its applicability in diverse contexts.
Broader impact
The advancements in TTS technology presented in this paper have significant implications for various applications, including real-time conversational agents, voice cloning, and multilingual speech synthesis. The model's ability to generate expressive and high-quality speech can enhance user experiences in virtual assistants, entertainment, and accessibility tools. The paper presents dots.tts, a state-of-the-art autoregressive TTS system that effectively addresses long-range error accumulation and improves synthesis quality through innovative methodologies. Its comprehensive evaluation and release of resources position it as a significant contribution to the field of machine learning and audio synthesis.
Author of this paper? Send a correction.