F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
Abstract
This paper introduces F5-TTS, a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT). Without requiring complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then the denoising is performed for speech generation, which was originally proved feasible by E2 TTS. However, the original design of E2 TTS makes it hard to follow due to its slow convergence and low robustness. To address these issues, we first model the input with ConvNeXt to refine the text representation, making it easy to align with the speech. We further propose an inference-time Sway Sampling strategy, which significantly improves our model's performance and efficiency. This sampling strategy for flow step can be easily applied to existing flow matching based models without retraining. Our design allows faster training and achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based TTS models. Trained on a public 100K hours multilingual dataset, our F5-TTS exhibits highly natural and expressive zero-shot ability, seamless code-switching capability, and speed control efficiency. We have released all codes and checkpoints to promote community development, at https://SWivid.github.io/F5-TTS/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of F5-TTS, a simplified yet effective non-autoregressive TTS system that leverages flow matching and a novel inference strategy to achieve high-quality speech synthesis. This work represents a meaningful advancement in the field of audio synthesis, addressing key challenges in TTS systems while promoting community engagement through open-source contributions.
Methodology
The paper presents a novel non-autoregressive text-to-speech system, F5-TTS, which utilizes flow matching with a Diffusion Transformer architecture. The methodology is innovative as it simplifies the TTS pipeline by removing complex components like duration models and phoneme alignment, instead using filler tokens and a refined text representation through ConvNeXt. The introduction of the Sway Sampling strategy at inference time is a significant enhancement, allowing for improved performance without retraining existing models. The approach is well-structured and addresses critical issues in previous models, particularly regarding robustness and efficiency.
Results & evaluation
The experiments are comprehensive, utilizing a large multilingual dataset of 100K hours and evaluating the model against various benchmarks. The results demonstrate strong performance in terms of word error rate (WER) and speaker similarity, with F5-TTS achieving competitive results compared to state-of-the-art models. The use of both objective metrics (WER, SIM) and subjective evaluations (CMOS, SMOS) adds rigor to the experimental validation.
Reproducibility
The authors have released their code and checkpoints, promoting transparency and reproducibility in their research. However, the paper could benefit from more detailed implementation specifics to facilitate easier replication of results by other researchers.
Limitations
The paper acknowledges limitations such as the model's inability to control fine-grained paralinguistic details like emotion and the challenge of efficiently handling longer mel spectrogram sequences compared to text inputs. These limitations highlight areas for future research and improvement.
Broader impact
The development of F5-TTS has significant implications for real-world applications in speech synthesis, particularly in multilingual contexts. The model's ability to generate high-quality, expressive speech with zero-shot capabilities could enhance accessibility in various domains, including education, entertainment, and communication technologies. However, ethical considerations regarding the potential misuse of TTS technology, such as voice spoofing, must be addressed. The main contribution of this paper is the introduction of F5-TTS, a simplified yet effective non-autoregressive TTS system that leverages flow matching and a novel inference strategy to achieve high-quality speech synthesis. This work represents a meaningful advancement in the field of audio synthesis, addressing key challenges in TTS systems while promoting community engagement through open-source contributions.
Curator note
Chen et al., CMU; flow matching TTS with rectified flow; state-of-the-art quality with fast inference
Author of this paper? Send a correction.