FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

Yi Ren, Chenxu Hu, ★ Xu Tan , ★ Sheng Zhao , ★ Tie-Yan Liu ...Full author list in the original paper.

Zhejiang University · arxiv

Affiliations: Zhejiang University

Abstract

Non-autoregressive text to speech (TTS) models such as FastSpeech can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

FastSpeech 2 presents a significant advancement in text-to-speech synthesis by simplifying the training process and improving voice quality through the incorporation of additional variance information. The methodology is innovative, and the experimental results demonstrate its effectiveness, making it a valuable contribution to the field of machine learning and audio processing.

Methodology

The methodology presented in FastSpeech 2 is robust, addressing key limitations of its predecessor, FastSpeech. The authors effectively simplify the training pipeline by eliminating the teacher-student distillation process and directly utilizing ground-truth targets, which enhances the model's performance and reduces training time. The introduction of additional variance information (duration, pitch, energy) as conditional inputs is a significant improvement that helps tackle the one-to-many mapping problem in TTS. The model architecture, which includes a variance adaptor and a direct waveform generation approach in FastSpeech 2s, is well-justified and innovative, making strides towards a fully end-to-end TTS system.

Results & evaluation

The experimental evaluation is comprehensive, utilizing the LJSpeech dataset to demonstrate the effectiveness of FastSpeech 2 and 2s. The authors provide clear comparisons of audio quality through mean opinion score (MOS) evaluations, showing that their models outperform previous systems, including autoregressive models. The reported training and inference speed improvements are substantial, with a 3x reduction in training time and significant speedups in inference, which are critical metrics for practical TTS applications.

Reproducibility

The paper includes sufficient details regarding the model architecture and experimental setup, including the dataset and evaluation metrics. However, the lack of a publicly available code repository limits reproducibility. While the authors provide audio samples and a demo URL, sharing the model code would greatly enhance the ability of other researchers to replicate and build upon this work.

Limitations

One limitation is the reliance on external tools for alignment and pitch extraction, which complicates the end-to-end nature of the system. Additionally, while the authors mention future work on eliminating the need for external alignment models, this remains a challenge for achieving a fully autonomous TTS system. The paper could also benefit from a more in-depth analysis of the limitations of FastSpeech 2 and 2s in terms of generalization across different languages and accents.

Broader impact

The advancements presented in this paper have significant implications for the TTS field, particularly in applications requiring high-quality, real-time speech synthesis, such as virtual assistants, audiobooks, and accessibility tools. The ability to generate natural-sounding speech quickly and efficiently could enhance user experiences across various platforms and services. FastSpeech 2 presents a significant advancement in text-to-speech synthesis by simplifying the training process and improving voice quality through the incorporation of additional variance information. The methodology is innovative, and the experimental results demonstrate its effectiveness, making it a valuable contribution to the field of machine learning and audio processing.

Curator note

Ren et al., Microsoft; duration/pitch/energy predictors; cleaner non-autoregressive TTS

Related later work

Author of this paper? Send a correction.