LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space
Abstract
We present LongCat-AudioDiT, a novel, non-autoregressive diffusion-based text-to-speech (TTS) model that achieves state-of-the-art (SOTA) performance. Unlike previous methods that rely on intermediate acoustic representations such as mel-spectrograms, the core innovation of LongCat-AudioDiT lies in operating directly within the waveform latent space. This approach effectively mitigates compounding errors and drastically simplifies the TTS pipeline, requiring only a waveform variational autoencoder (Wav-VAE) and a diffusion backbone. Furthermore, we introduce two critical improvements to the inference process: first, we identify and rectify a long-standing training-inference mismatch; second, we replace traditional classifier-free guidance with adaptive projection guidance to elevate generation quality. Experimental results demonstrate that, despite the absence of complex multi-stage training pipelines or high-quality human-annotated datasets, LongCat-AudioDiT achieves SOTA zero-shot voice cloning performance on the Seed benchmark while maintaining competitive intelligibility. Specifically, our largest variant, LongCat-AudioDiT-3.5B, outperforms the previous SOTA model (Seed-TTS), improving the speaker similarity (SIM) scores from 0.809 to 0.818 on Seed-ZH, and from 0.776 to 0.797 on Seed-Hard. Finally, through comprehensive ablation studies and systematic analysis, we validate the effectiveness of our proposed modules. Notably, we investigate the interplay between the Wav-VAE and the TTS backbone, revealing the counterintuitive finding that superior reconstruction fidelity in the Wav-VAE does not necessarily lead to better overall TTS performance. Code and model weights are released to foster further research within the speech community.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
LongCat-AudioDiT presents a significant advancement in text-to-speech synthesis through its innovative approach in the waveform latent space and the introduction of adaptive projection guidance. The comprehensive experimental results and the release of code and model weights contribute to its potential impact on the field, although further exploration of its limitations and broader applicability is warranted.
Methodology
The methodology presented in LongCat-AudioDiT is innovative, particularly in its non-autoregressive diffusion-based approach to text-to-speech synthesis. By operating directly in the waveform latent space rather than relying on intermediate representations like mel-spectrograms, the authors have simplified the TTS pipeline significantly. The introduction of adaptive projection guidance to replace traditional classifier-free guidance is a noteworthy advancement that enhances generation quality. The paper also addresses a critical training-inference mismatch, showcasing a thoughtful approach to improving model performance. Overall, the methodology is robust and well-structured, with clear innovations that set it apart from existing models.
Results & evaluation
The experimental evaluation is thorough, with the authors providing comprehensive results that demonstrate the effectiveness of LongCat-AudioDiT. The paper reports state-of-the-art performance on the Seed benchmark for zero-shot voice cloning, with significant improvements in speaker similarity scores. The use of ablation studies to validate the proposed modules adds credibility to the findings. However, the absence of high-quality human-annotated datasets may limit the generalizability of the results, although the authors mitigate this by achieving competitive intelligibility.
Reproducibility
The authors mention that code and model weights are released, which is a positive aspect for reproducibility. However, the paper could benefit from more detailed implementation guidelines and hyperparameter settings to facilitate easier replication of the results by other researchers.
Limitations
One limitation identified is the reliance on a single benchmark (Seed) for evaluation, which may not fully capture the model's performance across diverse TTS tasks. Additionally, the findings regarding the Wav-VAE's reconstruction fidelity not correlating with TTS performance could indicate a need for further exploration into the underlying mechanisms affecting performance.
Broader impact
The potential applications of LongCat-AudioDiT are significant, particularly in areas requiring high-fidelity speech synthesis, such as virtual assistants, audiobooks, and voice cloning technologies. The model's ability to operate without complex multi-stage training pipelines could democratize access to high-quality TTS systems, fostering innovation in various industries. LongCat-AudioDiT presents a significant advancement in text-to-speech synthesis through its innovative approach in the waveform latent space and the introduction of adaptive projection guidance. The comprehensive experimental results and the release of code and model weights contribute to its potential impact on the field, although further exploration of its limitations and broader applicability is warranted.
Author of this paper? Send a correction.