Bayesian Speech synthesizers Can Learn from Multiple Teachers
Abstract
Codec-based text-to-speech (TTS) models have recently gained traction for their efficiency and strong performance in voice cloning. However, codec-based TTS faces limitations due to the challenges of pretraining robust speech codecs and the quality degradation introduced by quantization errors. Emerging evidence suggests that continuous-valued generative models can alleviate these issues and serve as a promising alternative. Yet, effectively modelling diverse speech patterns and developing reliable sampling strategies for continuous-valued autoregressive (AR) TTS remains underexplored. In this work, we propose BELLE, Bayesian evidential learning with language modelling for TTS, a novel continuous-valued AR framework that directly predicts mel-spectrograms from textual input. BELLE treats each mel-spectrogram frame as a Gaussian distribution sampled from a learned hyper distribution, enabling principled uncertainty estimation, particularly in scenarios with parallel data (i.e., one text-audio prompt paired with multiple speech samples). To obtain such data, diverse speech samples are synthesized using multiple pre-trained TTS models given the same text-audio prompts, which are distilled into BELLE via Bayesian evidential learning. Experimental results indicate that BELLE demonstrates highly competitive performance compared with the current best open-source TTS models, even though BELLE is trained on a large amount of synthetic data and uses only approximately one-tenth of their training data. Audio samples generated by BELLE are available at https://belletts.github.io/Belle/. The code, checkpoints, and synthetic data will be released after the paper is accepted.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of BELLE, a continuous-valued autoregressive TTS model that leverages Bayesian evidential learning and multi-teacher knowledge distillation, achieving competitive performance with significantly less training data. This work represents a notable advancement in the field of text-to-speech synthesis, addressing key challenges in audio generation and uncertainty estimation.
Methodology
The paper introduces BELLE, a novel continuous-valued autoregressive TTS model utilizing Bayesian evidential learning for uncertainty estimation in mel-spectrogram prediction. The methodology is well-structured, leveraging multi-teacher knowledge distillation to enhance training efficiency and output quality. The approach of treating each mel-spectrogram frame as a Gaussian distribution sampled from a hyper-distribution is innovative and addresses the limitations of existing codec-based TTS models. The integration of Bayesian methods into TTS is a significant advancement, providing a principled framework for uncertainty quantification and improving the robustness of the model.
Results & evaluation
The experimental setup is thorough, utilizing a substantial dataset derived from Librispeech and augmented with synthetic samples from multiple pre-trained TTS models. The results demonstrate that BELLE achieves competitive performance against state-of-the-art models while using significantly less training data. The evaluation metrics, including subjective (MOS, SMOS) and objective (WER, SIM) measures, provide a comprehensive assessment of the model's capabilities. The findings indicate that BELLE not only maintains high audio quality but also enhances speaker similarity and diversity in the generated speech.
Reproducibility
The paper includes a reproducibility statement, indicating that code, checkpoints, and synthetic data will be made available upon acceptance. The detailed descriptions of model configurations, training parameters, and experimental setups enhance the reproducibility of the results. However, the reliance on multiple external TTS models for data augmentation may complicate replication efforts if those models are not readily accessible.
Limitations
The study primarily focuses on English speech data, limiting its applicability to multilingual contexts. Additionally, the real-time factor (RTF) of BELLE is still higher compared to some recent models, indicating potential latency issues in real-time applications. The authors acknowledge these limitations and suggest future work to extend BELLE's capabilities to other languages and improve latency.
Broader impact
BELLE has significant potential applications in various domains, including assistive communication technologies, personalized education platforms, and human-machine interaction systems. However, the paper also highlights ethical concerns regarding the misuse of TTS technologies, such as impersonation and identity fraud, emphasizing the need for responsible use and regulatory oversight. The main contribution of this paper is the introduction of BELLE, a continuous-valued autoregressive TTS model that leverages Bayesian evidential learning and multi-teacher knowledge distillation, achieving competitive performance with significantly less training data. This work represents a notable advancement in the field of text-to-speech synthesis, addressing key challenges in audio generation and uncertainty estimation.
Author of this paper? Send a correction.