PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis
Abstract
Building state-of-the-art text-to-speech (TTS) systems typically demands millions of hours of proprietary data and complex multi-stage architectures, creating substantial barriers for resource-constrained research teams. In this report, we present PilotTTS, a lightweight autoregressive TTS system that achieves competitive performance through minimalist architecture and rigorous data engineering. PilotTTS is trained on only 200K hours of data processed entirely with open-source tools. Specifically, our contributions are: (1) a reproducible multi-stage data processing pipeline covering quality assessment, label annotation, and filtering, and (2) a compact model architecture that employs Q-Former-based conditioning to decouple speaker identity from speaking style via cross-sample paired training. Within a unified framework, PilotTTS supports zero-shot voice cloning, emotion synthesis (11 categories), paralinguistic synthesis (4 categories), and Chinese dialect synthesis (14 dialects). On the Seed-TTS Eval benchmark, PilotTTS achieves the lowest WER of 1.50% on test-en, a CER of 0.87% on test-zh, and the highest speaker similarity on both test sets (0.862 and 0.815), outperforming systems trained on significantly larger datasets. We release the complete data pipeline recipe, pretrained weights, and code at https://github.com/AMAPVOICE/PilotTTS.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of PilotTTS, a lightweight and competitive TTS system that leverages rigorous data engineering and a disciplined modular architecture to achieve state-of-the-art performance with significantly less training data than existing systems. This work is significant as it addresses the barriers faced by resource-constrained teams in the field of speech synthesis, providing a practical solution that maintains high performance while promoting reproducibility and accessibility.
Methodology
The methodology is robust, featuring a well-structured multi-stage data processing pipeline that enhances data quality and a compact autoregressive architecture that effectively decouples speaker identity from style. The use of Q-Former-based conditioning and cross-sample paired training is innovative and addresses common challenges in TTS systems.
Results & evaluation
The experiments are comprehensive, utilizing the Seed-TTS Eval benchmark to demonstrate superior performance in terms of WER, CER, and speaker similarity. The inclusion of human evaluations for emotion control and paralinguistic synthesis adds depth to the assessment of the system's capabilities.
Reproducibility
The paper emphasizes reproducibility by providing a complete data processing pipeline built from publicly available tools, along with pretrained weights and code. This transparency enhances the likelihood of other researchers replicating the results.
Limitations
The paper acknowledges limitations such as insufficient explicit style modeling and the constraints of single-codebook quantization, which may hinder performance in more complex scenarios. Additionally, the reliance on mel-spectrograms could introduce reconstruction artifacts.
Broader impact
The potential applications of PilotTTS are significant, particularly for resource-constrained teams seeking to develop competitive TTS systems. Its modular approach and open-source nature could democratize access to high-quality speech synthesis technology. The main contribution of this paper is the introduction of PilotTTS, a lightweight and competitive TTS system that leverages rigorous data engineering and a disciplined modular architecture to achieve state-of-the-art performance with significantly less training data than existing systems. This work is significant as it addresses the barriers faced by resource-constrained teams in the field of speech synthesis, providing a practical solution that maintains high performance while promoting reproducibility and accessibility.
Author of this paper? Send a correction.