FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation
Abstract
Recent progress in speech dialogue systems requires Text-to-Speech (TTS) models to be faster and more responsive. Modern speech dialogue systems impose two primary requirements on TTS models: low latency and support for streaming inputs and outputs. However, most existing single-codebook LLM-based TTS methods rely on multi-stage pipelines that lack native streaming capabilities. These systems typically suffer from high end-to-end latency due to slow autoregressive prediction and multi-step flow matching. To address these limitations, we propose FlashTTS, an open-source and low-latency streaming TTS framework. FlashTTS introduces a lagged multi-track architecture that natively processes streaming text and speech inputs, thereby eliminating the need for sentence-level buffering. To accelerate acoustic generation, we integrate parallel Multi-Token Prediction (MTP) with an X-pred mean flow matching decoder. This configuration achieves high-fidelity token-to-mel generation in exactly two function evaluations (2-NFE). By jointly optimizing input processing and decoding efficiency, FlashTTS offers a practical foundation for real-time speech dialogue systems. Experiments show that FlashTTS substantially reduces First-Packet Latency to 325ms compared to robust streaming baselines, all while preserving strong zero-shot voice cloning and cross-lingual intelligibility. Speech samples are available. The model code and checkpoints will be released as open source.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
FlashTTS represents a substantial advancement in low-latency TTS systems, effectively addressing the challenges of real-time speech synthesis through innovative architectural design and optimization strategies. The comprehensive evaluation and commitment to open-source release further solidify its potential impact on the field of speech processing.
Methodology
The methodology presented in FlashTTS is innovative, focusing on a lagged multi-track architecture that supports real-time streaming TTS. The integration of Multi-Token Prediction (MTP) with an X-pred mean flow matching decoder is a significant advancement, allowing for high-fidelity audio generation with minimal latency. The paper effectively addresses the limitations of existing TTS systems by eliminating sentence-level buffering and optimizing both input processing and decoding efficiency. The design choices are well-justified, and the architecture is clearly articulated, making it a robust contribution to the field.
Results & evaluation
The experimental evaluation is thorough, utilizing a substantial dataset of 300,000 hours of speech data and multiple benchmarks for performance assessment. The results demonstrate a significant reduction in First-Packet Latency (FPL) and First-Token Latency (FTL) compared to existing models, while maintaining strong performance in terms of intelligibility and quality across multiple languages. The use of both objective metrics (WER, SIM) and subjective evaluations (CMOS) provides a comprehensive view of the model's performance.
Reproducibility
The paper provides sufficient details regarding the model architecture, training setup, and evaluation metrics, which enhances reproducibility. The authors also commit to releasing the model code and checkpoints as open source, further supporting the community's ability to replicate and build upon their work.
Limitations
While the paper presents a strong case for the effectiveness of FlashTTS, it does not thoroughly address potential limitations in terms of scalability or the impact of varying input complexities on performance. Additionally, the reliance on a specific architecture (Qwen2-0.5B) may limit generalizability to other frameworks or applications.
Broader impact
The implications of FlashTTS are significant, particularly for real-time speech dialogue systems, which are increasingly important in various applications, including virtual assistants, customer service bots, and interactive gaming. The ability to generate high-fidelity speech with low latency could enhance user experiences across these domains, making technology more accessible and effective. FlashTTS represents a substantial advancement in low-latency TTS systems, effectively addressing the challenges of real-time speech synthesis through innovative architectural design and optimization strategies. The comprehensive evaluation and commitment to open-source release further solidify its potential impact on the field of speech processing.
Author of this paper? Send a correction.