Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
Abstract
Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in \url{https://voicebox.metademolab.com}.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents Voicebox, a groundbreaking model for text-guided multilingual speech generation that significantly advances the state of the art in generative speech modeling. The innovative methodology, extensive experimental validation, and potential for broad applications underscore its importance in the field of machine learning and audio processing.
Methodology
The methodology presented in this paper is innovative, leveraging a non-autoregressive flow-matching model trained on a large dataset of over 50K hours of speech. The model's ability to infill speech based on both audio context and text input is a significant advancement in generative speech models. The use of in-context learning allows for task generalization without the need for extensive labeled data, which is a notable departure from traditional approaches that rely heavily on labeled datasets. The introduction of a flow-matching objective with optimal transport paths is a novel contribution that enhances the efficiency and scalability of the model.
Results & evaluation
The experimental evaluation is robust, with comprehensive comparisons against state-of-the-art models such as VALL-E and YourTTS across multiple tasks, including zero-shot TTS, noise removal, and content editing. The results demonstrate significant improvements in intelligibility and audio similarity, with quantitative metrics like word error rates and audio similarity scores clearly indicating the model's superiority. The inclusion of diverse applications and the ability to generate high-quality speech across multiple languages further validate the model's effectiveness.
Reproducibility
The paper provides detailed descriptions of the training setup, model architecture, and evaluation metrics, which supports reproducibility. However, the absence of a publicly available code repository may hinder full reproducibility for other researchers. The authors do mention the use of standard datasets and established metrics, which aids in comparing results with future work.
Limitations
While the model shows promising results, it is important to note that it has not been tested on all possible speech styles or in all acoustic conditions. The reliance on large datasets may also introduce biases inherent in the training data, which could affect the model's performance in real-world applications. Additionally, the ethical implications of generating speech in the style of arbitrary individuals are acknowledged but not deeply explored.
Broader impact
The potential applications of this work are vast, ranging from enhancing accessibility through improved TTS systems to creating more engaging virtual assistants and content generation tools. The ability to generate high-quality, contextually relevant speech could revolutionize industries such as entertainment, education, and customer service. However, the ethical considerations surrounding the misuse of such technology for impersonation or misinformation must be addressed. This paper presents Voicebox, a groundbreaking model for text-guided multilingual speech generation that significantly advances the state of the art in generative speech modeling. The innovative methodology, extensive experimental validation, and potential for broad applications underscore its importance in the field of machine learning and audio processing.
Curator note
Le et al., Meta; flow-matching TTS at scale; in-context learning for voice styles
Author of this paper? Send a correction.