OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models
Abstract
We present OmniVoice, a massive multilingual zero-shot text-to-speech (TTS) model that scales to over 600 languages. At its core is a novel diffusion language model-style discrete non-autoregressive (NAR) architecture. Unlike conventional discrete NAR models that suffer from performance bottlenecks in complex two-stage (text-to-semantic-to-acoustic) pipelines, OmniVoice directly maps text to multi-codebook acoustic tokens. This simplified approach is facilitated by two key technical innovations: (1) a full-codebook random masking strategy for efficient training, and (2) initialization from a pre-trained LLM to ensure superior intelligibility. By leveraging a 581k-hour multilingual dataset curated entirely from open-source data, OmniVoice achieves the broadest language coverage to date and delivers state-of-the-art performance across Chinese, English, and diverse multilingual benchmarks. Our code and pre-trained models are publicly available at https://github.com/k2-fsa/OmniVoice.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of OmniVoice, a multilingual zero-shot TTS model that achieves state-of-the-art performance across over 600 languages by leveraging innovative methodologies in model architecture and training. This work represents a significant advancement in the field of text-to-speech synthesis, particularly in addressing the challenges of low-resource language support and intelligibility in generated speech.
Methodology
The methodology presented in OmniVoice is innovative, utilizing a diffusion language model-style discrete non-autoregressive architecture that simplifies the text-to-speech pipeline by directly mapping text to acoustic tokens. The introduction of full-codebook random masking and initialization from a pre-trained LLM are significant advancements that enhance training efficiency and intelligibility. The approach is well-structured, leveraging a large multilingual dataset to achieve broad language coverage and high-quality speech synthesis.
Results & evaluation
The experimental evaluation is comprehensive, covering multiple benchmarks across different languages, including both objective and subjective metrics. The paper demonstrates state-of-the-art performance in intelligibility and speaker similarity, validating the model's effectiveness through rigorous testing. The results indicate that OmniVoice outperforms existing models, particularly in low-resource languages, which is a notable achievement.
Reproducibility
The paper provides sufficient implementation details, including model architecture, training protocols, and evaluation metrics, which should facilitate reproducibility. The availability of code and pre-trained models on GitHub further supports this goal. However, the lack of specific citations for some methodologies could hinder full reproducibility.
Limitations
While the model shows promise, it may still face challenges with certain low-resource languages, as indicated by varying performance metrics. Additionally, the reliance on open-source datasets may introduce variability in data quality, potentially affecting the model's robustness. The paper does not address potential biases in the training data or the implications of using a large multilingual dataset.
Broader impact
OmniVoice has the potential to significantly impact the accessibility of text-to-speech technology across a wide range of languages, particularly for low-resource languages that are often neglected in mainstream TTS systems. This could enhance communication and technology access for speakers of these languages, promoting inclusivity in AI applications. The main contribution of this paper is the introduction of OmniVoice, a multilingual zero-shot TTS model that achieves state-of-the-art performance across over 600 languages by leveraging innovative methodologies in model architecture and training. This work represents a significant advancement in the field of text-to-speech synthesis, particularly in addressing the challenges of low-resource language support and intelligibility in generated speech.
Author of this paper? Send a correction.