DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
Abstract
Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness by propagating local errors and hallucinations. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, as the full input text is available before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a $1/t$-weighted training objective and a time-shifted inference schedule that defer low-confidence positions to later steps. Trained on only $585$ hours of LibriTTS, DELTA-TTS achieves a $\textbf{1.75}\%$ WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens $\textbf{3.3}\times$ faster. Further analysis shows that DELTA-TTS produces sharper text--speech alignment, increases overall decoding confidence, and mitigates hallucinations observed in AR generation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
DELTA-TTS effectively bridges the gap between autoregressive and diffusion-based speech synthesis by adapting a pretrained AR model into a discrete diffusion language model with speech-aware modifications, achieving state-of-the-art speed-quality trade-offs and robustness against hallucinations.
Methodology
The paper proposes DELTA-TTS, a framework that converts a pretrained Autoregressive (AR) Text-to-Speech (TTS) model into a discrete Diffusion Language Model (dLLM). The core methodological contribution lies in adapting the AR-to-dLLM conversion technique, previously established for text, to the speech domain. Key innovations include: 1) Replacing causal attention with bidirectional attention to allow access to future context; 2) Introducing a Conformer-style convolution module to inject local acoustic inductive biases that bidirectional self-attention lacks; 3) Utilizing a $1/t$-weighted training objective and a time-shifted inference schedule to defer low-confidence token predictions to later steps. The approach is parameter-efficient, using LoRA adapters on top of a frozen CosyVoice3 backbone. The methodology is sound and addresses a known bottleneck in AR TTS (inference speed and error propagation) by leveraging the parallel generation capabilities of diffusion models.
Results & evaluation
The evaluation is comprehensive, comparing DELTA-TTS against a wide range of state-of-the-art AR and Non-Autoregressive (NAR) baselines, including Seed-TTS, CosyVoice series, FireRedTTS, and F5-TTS. The primary metric is Word Error Rate (WER) on Seed-TTS test-en and LibriSpeech-PC test-clean. DELTA-TTS achieves a WER of 1.75% on Seed-TTS, outperforming the AR backbone (CosyVoice3, 2.02%) and all listed NAR baselines. It also demonstrates superior Real-Time Factor (RTF), achieving a 3.3x speedup. The paper includes ablation studies validating the necessity of the convolution module and the time-shifted schedule. Furthermore, it provides qualitative analysis showing sharper text-speech alignment and mitigation of hallucinations (catastrophic failures) observed in the AR baseline. The use of both objective metrics (WER, SIM, UTMOS) and subjective evaluations (CMOS, SMOS) strengthens the claims.
Reproducibility
The paper provides sufficient implementation details for reproduction. It specifies the backbone (CosyVoice3, Qwen2-0.5B), the adapter size (94M parameters), training data (585 hours LibriTTS), optimizer settings (AdamW, lr $10^{-4}$), and specific architectural choices (kernel size 31 for convolution). The training data size is small and publicly available, facilitating reproducibility. However, the code is not explicitly linked in the provided text, and the "rule-based length" prediction method is described but might require careful implementation to match results. The reliance on specific baselines' checkpoints (e.g., CosyVoice3) is noted, which is standard but requires access to those resources.
Limitations
The primary limitation is the training data scale (585 hours), which is significantly smaller than the multi-hundred-thousand-hour datasets used by many baselines. While this highlights the efficiency of the adaptation, it may limit the model's generalization to out-of-distribution speakers or languages compared to larger models. The paper acknowledges this and notes the model is English-only. Additionally, the target length prediction is rule-based, which is a known weakness in dLLM-based TTS systems. The speedup is measured on token generation only; end-to-end latency including the flow-matching decoder is not fully detailed, though the decoder is frozen and shared.
Broader impact
This work contributes to the democratization of high-quality TTS by enabling faster inference and potentially lower training costs through efficient adaptation. It reduces the computational barrier for deploying TTS systems. The mitigation of hallucinations improves reliability. However, like all generative audio technologies, it carries risks related to misuse for deepfakes, though the paper frames it as academic research. The ability to generate speech faster and more robustly has positive implications for assistive technologies and interactive voice assistants. DELTA-TTS effectively bridges the gap between autoregressive and diffusion-based speech synthesis by adapting a pretrained AR model into a discrete diffusion language model with speech-aware modifications, achieving state-of-the-art speed-quality trade-offs and robustness against hallucinations.
Author of this paper? Send a correction.