LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling
Abstract
Large language model (LLM)-based text-to-speech (TTS) systems achieve remarkable naturalness via autoregressive (AR) decoding, but require N sequential steps to generate N speech tokens. We present LLaDA-TTS, which replaces the AR LLM with a masked diffusion model that completes generation in a fixed number of parallel steps, decoupling inference latency from sequence length. Remarkably, using only 50 hours of fine-tuning data, we successfully transfer a pretrained AR checkpoint to the masked diffusion paradigm via bidirectional attention. At 64 steps, LLaDA-TTS achieves 0.98% CER (zh) and 1.96% WER (en) on Seed-TTS-Eval, matching the original CosyVoice 3 baseline performance while delivering a 2x LLM-stage speedup--a notable acceleration achieved despite the absence of KV cache, an optimization the AR baseline heavily relies on. Beyond acceleration, the bidirectional architecture naturally enables zero-shot speech editing--including word-level insertion, deletion, and substitution--without any additional training. Theoretically, we prove that AR-pretrained weights are near-optimal for bidirectional masked prediction under the locality property of acoustic tokens, explaining this rapid convergence. This general method modifies only the attention mask and objective, applying seamlessly to any LLM-based AR TTS system. Code and audio samples will be available at https://deft-piroshki-b652b5.netlify.app/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of LLaDA-TTS, a novel masked diffusion model for TTS that decouples inference latency from sequence length while enabling zero-shot speech editing capabilities. This work represents a significant advancement in the efficiency and functionality of TTS systems, with implications for both research and practical applications in the field.
Methodology
The paper introduces LLaDA-TTS, which innovatively replaces the autoregressive (AR) text-to-speech (TTS) model with a masked diffusion model. This approach allows for parallel generation of speech tokens, significantly reducing inference latency. The methodology is well-grounded in theoretical analysis, particularly the proof that AR-pretrained weights are near-optimal for bidirectional masked prediction. The architecture is adaptable to various LLM-based TTS systems, enhancing its applicability. The use of a bidirectional attention mechanism enables zero-shot speech editing, which is a notable advancement in TTS technology. However, the reliance on a specific initialization and the necessity of determining output length in advance may limit flexibility.
Results & evaluation
The experiments conducted on the Seed-TTS-Eval benchmark demonstrate the efficacy of LLaDA-TTS, achieving competitive character and word error rates while maintaining a significant speed advantage over traditional AR models. The results are well-documented, with comparisons against established baselines, and the paper provides a thorough analysis of the performance metrics. The findings on emergent behaviors, such as AR-like unmasking and attention alignment, are particularly noteworthy, showcasing the model's ability to leverage learned structures from pretraining effectively.
Reproducibility
The paper mentions that code and audio samples will be available, which is a positive aspect for reproducibility. However, details on the training process, hyperparameters, and specific configurations used in experiments could be more explicitly outlined to enhance reproducibility further.
Limitations
The primary limitations include the requirement to specify output length in advance and the non-streaming nature of the current implementation. These factors may hinder real-time applications and flexibility in various use cases. Additionally, while the model shows promise in zero-shot editing, the effectiveness of this feature across diverse scenarios remains to be fully validated.
Broader impact
LLaDA-TTS has the potential to significantly impact the field of speech synthesis by providing a faster and more flexible approach to generating high-quality speech. The ability to perform zero-shot editing could revolutionize applications in voiceovers, audiobooks, and interactive voice response systems. The findings may also inspire further research into the integration of diffusion models in other generative tasks within audio and beyond. The main contribution of this paper is the introduction of LLaDA-TTS, a novel masked diffusion model for TTS that decouples inference latency from sequence length while enabling zero-shot speech editing capabilities. This work represents a significant advancement in the efficiency and functionality of TTS systems, with implications for both research and practical applications in the field.
Author of this paper? Send a correction.