DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation
Abstract
Recent attempts to interleave autoregressive (AR) sketchers with diffusion-based refiners over continuous speech representations have shown promise, but they remain brittle under distribution shift and offer limited levers for controllability. We introduce DISTAR, a zero-shot text-to-speech framework that operates entirely in a discrete residual vector quantization (RVQ) code space and tightly couples an AR language model with a masked diffusion model, without forced alignment or a duration predictor. Concretely, DISTAR drafts block-level RVQ tokens with an AR language model and then performs parallel masked-diffusion infilling conditioned on the draft to complete the next block, yielding long-form synthesis with blockwise parallelism while mitigating classic AR exposure bias. The discrete code space affords explicit control at inference: DISTAR produces high-quality audio under both greedy and sample-based decoding using classifier-free guidance, supports trade-offs between robustness and diversity, and enables variable bit-rate and controllable computation via RVQ layer pruning at test time. Extensive experiments and ablations demonstrate that DISTAR surpasses state-of-the-art zero-shot TTS systems in robustness, naturalness, and speaker/style consistency, while maintaining rich output diversity. Audio samples are provided on https://anonymous.4open.science/w/DiSTAR_demo.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces DiSTAR, a zero-shot TTS framework that effectively combines autoregressive and diffusion modeling in a discrete code space, achieving state-of-the-art performance while addressing critical issues in previous approaches. The innovative methodology and comprehensive experimental validation position this work as a significant contribution to the field of speech synthesis.
Methodology
The paper presents a novel approach to zero-shot text-to-speech (TTS) synthesis by integrating an autoregressive (AR) language model with a masked diffusion model, operating entirely within a discrete residual vector quantization (RVQ) code space. This method effectively addresses the limitations of previous continuous-latent models, such as exposure bias and robustness under distribution shifts. The architecture is designed to allow for block-level parallelism and intra-frame depth modeling, which enhances the quality and coherence of the generated speech. The use of RVQ enables explicit control during inference, a significant improvement over traditional methods that rely on continuous representations.
Results & evaluation
The authors conducted extensive experiments on standard zero-shot TTS benchmarks, demonstrating that their model outperforms existing state-of-the-art systems in terms of robustness, naturalness, and speaker/style consistency. The evaluation metrics included both objective measures (like Word Error Rate and speaker similarity) and subjective assessments (mean opinion scores), providing a comprehensive view of the model's performance. The results indicate a strong scaling behavior with model capacity, suggesting that the proposed method is not only effective but also scalable.
Reproducibility
The paper includes detailed implementation specifics, including the architecture, training procedures, and optimization techniques used. It specifies the use of 64 NVIDIA A100 GPUs and provides insights into the training dataset and evaluation metrics. However, the absence of a publicly available code repository limits the reproducibility of the results, as other researchers may find it challenging to replicate the experiments without access to the code.
Limitations
The study acknowledges that its results are based solely on an English corpus of approximately 50k hours, which may limit the generalizability of the findings to multilingual or multi-style settings. Additionally, while the model shows promise in robustness and quality, the potential for misuse in generating impersonations and disinformation is a significant ethical concern that the authors briefly address.
Broader impact
The advancements in high-fidelity zero-shot TTS have significant implications for accessibility, education, and creative industries. However, the ability to closely mimic speaker timbre raises ethical issues, including risks of impersonation and misuse. The authors advocate for responsible deployment practices, including consent-first policies and audio watermarking to mitigate potential harms. The paper introduces DiSTAR, a zero-shot TTS framework that effectively combines autoregressive and diffusion modeling in a discrete code space, achieving state-of-the-art performance while addressing critical issues in previous approaches. The innovative methodology and comprehensive experimental validation position this work as a significant contribution to the field of speech synthesis.
Author of this paper? Send a correction.