Luna-TTS Family Technical Report
Abstract
Modern text-to-speech (TTS) is dominated by autoregressive (AR) codec language models, whose left-to-right decoding brings latency that grows with utterance length, error accumulation along the committed prefix, and an artificial generation order imposed on the Residual Vector Quantization (RVQ) token grid. We propose Luna-TTS Family, diffusion-language-model-based TTS systems pretrained on 1 million hours of speech across Chinese, English, Japanese, and Korean. The family is built by progressive adaptation of a pretrained AR text LLM, from causal to bidirectional and finally to block-causal attention, and comprises two variants sharing a single tokenizer, data pipeline, and 0.6B backbone lineage. Luna-TTS is fully non-autoregressive: it generates the entire RVQ token grid in a fixed number of parallel refinement steps, with zero-shot voice cloning and speech editing arising natively as infilling. Luna-TTS Realtime, derived by continual training, is autoregressive over blocks of 32 codec frames (1.28s) while denoising each block in parallel; it supports KV-cached blockwise generation and incremental audio delivery, achieving an end-to-end RTF of 0.0240 and 41.6 ms local first-block latency under the warmed serving protocol. An annealed fine-tuning stage adds explicit control over emotion and non-verbal vocalizations (NVVs), and a reinforcement-learning stage applies GRPO with policy ratios computed over the realized denoising trajectory. On Seed-TTS-Eval, Luna-TTS achieves the best results on all four metrics among compared open-source and commercial systems (0.73 CER / 79.7 SIM on test-zh, 1.49 WER / 76.8 SIM on test-en); on the harder in-the-wild CV3-Eval, it posts the lowest Mandarin and English error rates in our comparison. Against leading commercial systems, it achieves the best results on most objective, model-based, and human-rated metrics for NVV and emotion control.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Luna-TTS introduces a scalable, block-diffusion-based TTS framework that achieves state-of-the-art quality and latency by adapting pretrained LLMs through progressive architectural changes and RL post-training, effectively resolving the latency-quality trade-off inherent in previous diffusion TTS systems.
Methodology
The paper proposes a novel architecture for Text-to-Speech (TTS) by adapting Large Language Models (LLMs) into Diffusion Language Models (dLLMs). Specifically, it introduces a "block-causal" attention mechanism that allows for streaming generation while retaining the parallel denoising benefits of diffusion models. The approach involves a progressive adaptation from causal to bidirectional and finally to block-causal attention on a pretrained AR text LLM. A key technical contribution is the application of Group Relative Policy Optimization (GRPO) directly over the realized denoising trajectory, optimizing for content correctness and speaker similarity. The use of a semantic-distilled RVQ tokenizer where the first codebook is anchored to linguistic content is a significant methodological detail that facilitates the diffusion process. The transition from fully parallel masked diffusion to block-autoregressive diffusion for latency reduction is a well-reasoned engineering solution to a known bottleneck in diffusion-based generation.
Results & evaluation
The evaluation is comprehensive, covering both objective metrics (CER, WER, SIM) and subjective/human-rated metrics for emotion and non-verbal vocalization (NVV) control. The paper claims state-of-the-art performance on Seed-TTS-Eval and CV3-Eval, outperforming both open-source and leading commercial systems. The inclusion of a "warmed serving protocol" to measure real-time factor (RTF) and first-block latency adds practical relevance to the experimental setup. The comparison against commercial baselines strengthens the claim of production-readiness. However, the specific numerical results are redacted in the provided text, preventing a precise verification of the magnitude of improvement, though the qualitative claims are strong.
Reproducibility
The paper provides detailed descriptions of the architecture, including the tokenizer design, the masked diffusion formulation, and the training schedule. The mention of "0.6B backbone lineage" and "1 million hours of speech" provides scale context. However, the specific hyperparameters for the GRPO stage and the exact implementation details of the block-causal attention masking are likely critical for reproduction and may be subject to interpretation. The lack of publicly released code or weights (implied by "none" for URLs) significantly hinders immediate reproducibility for the broader community, although the technical report format suggests a high level of detail.
Limitations
The primary limitation is the reliance on a specific RVQ tokenizer and the assumption that the semantic anchoring of the first codebook generalizes well across all languages and domains. The block-diffusion approach, while reducing latency, still requires multiple denoising steps per block, which may not be as fast as single-step AR decoding in extreme low-latency scenarios. Furthermore, the complexity of training a diffusion LLM with RL post-training is significantly higher than standard AR TTS training, potentially limiting accessibility. The paper does not extensively discuss the failure modes of the NVV control or the robustness of the emotion conditioning under adversarial text inputs.
Broader impact
This work represents a significant step towards making diffusion-based generative models viable for real-time, production-grade audio applications. By bridging the gap between the quality of diffusion models and the latency requirements of streaming TTS, it expands the toolkit available for developers building voice assistants, gaming NPCs, and accessibility tools. The emphasis on expressive control (emotion, NVVs) aligns with the growing demand for more natural and engaging human-computer interaction. However, the potential for misuse in deepfake generation remains a concern, necessitating robust safety measures in deployment. Luna-TTS introduces a scalable, block-diffusion-based TTS framework that achieves state-of-the-art quality and latency by adapting pretrained LLMs through progressive architectural changes and RL post-training, effectively resolving the latency-quality trade-off inherent in previous diffusion TTS systems.
Author of this paper? Send a correction.