One-Step Token-to-Waveform Generation with MeanFlow in Latent Space
Abstract
Neural audio codecs are central to modern LLM-based Text-to-Speech (TTS) and multimodal systems. As low-bitrate semantic codecs gain prominence, the Token-to-Waveform (Token2Wav) decoder becomes a bottleneck determining both perceptual quality and system efficiency. Conventional multi-step flow-matching decoders offer superior quality but suffer from high inference latency due to iterative sampling, creating a severe quality-speed trade-off. In this paper, we propose a novel Token2Wav architecture that overcomes this limitation by applying MeanFlow in a highly compressed latent space. By modeling the average velocity rather than the instantaneous velocity field, MeanFlow enables true one-step generation. Operating in the latent domain mitigates the memory and stability issues of waveform-level flows, yielding up to a 17$\times$ improvement in Real-Time Factor (RTF) compared to multi-step baselines with negligible quality degradation. Furthermore, we introduce refinement strategies that mitigate latent mismatch, including decoder-only fine-tuning with the MeanFlow generator frozen and end-to-end joint fine-tuning, improving fidelity without increasing inference-time cost. Code and demo are publicly available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant advancement in low-latency neural audio synthesis by successfully adapting MeanFlow to a latent space for one-step Token-to-Waveform generation, achieving a substantial reduction in inference latency while maintaining high perceptual quality through innovative refinement strategies.
Methodology
The paper proposes a novel architecture for Token-to-Waveform (Token2Wav) generation by combining MeanFlow (a one-step flow matching variant) with a latent space representation. The core innovation lies in applying MeanFlow in a compressed latent space rather than waveform space, which addresses the stability and memory issues typically associated with one-step generation at high resolutions. The methodology includes a lightweight VAE for latent encoding/decoding and a DiT-1D model for the one-step latent generation. Crucially, the authors address the "latent mismatch" problem—where generated latents deviate from the VAE's training distribution—through two refinement strategies: decoder-only fine-tuning and end-to-end joint fine-tuning. This is a technically sound approach that effectively bridges the gap between the efficiency of autoregressive/LLM-based token generation and the quality of continuous flow-based vocoders. The use of MeanFlow to eliminate iterative sampling is a significant methodological contribution to low-latency audio synthesis.
Results & evaluation
The experimental setup is rigorous, utilizing LibriTTS for training and LibriSpeech test-clean for evaluation. The authors compare their method against CosyVoice2's Token2Wav module, a strong multi-step flow baseline. They report Real-Time Factor (RTF), Word Error Rate (WER), Speaker Similarity (SpkSim), and perceptual metrics (UTMOS, MOS). The results demonstrate a substantial speedup (17x RTF improvement) with negligible degradation in quality metrics. The ablation studies are particularly valuable, analyzing the impact of latent dimensionality, model capacity, and the specific refinement strategies. The finding that larger models do not necessarily perform better in one-step generation is an insightful empirical observation. The inclusion of both objective and subjective metrics strengthens the evaluation.
Reproducibility
The paper provides clear descriptions of the model architectures, loss functions, and training procedures. The code and demo are publicly available on GitHub, which significantly enhances reproducibility. The authors specify the tokenization scheme (CosyVoice2 tokenizer), speaker embedding extractor (CAM++), and evaluation protocols. The detailed breakdown of RTF and the specific hyperparameters for the VAE and DiT models allow for faithful reproduction of the results.
Limitations
The primary limitation is the reliance on a pre-trained VAE, which introduces a fixed bottleneck in terms of information loss during encoding. While the refinement strategies mitigate this, they do not eliminate the fundamental trade-off between compression ratio and fidelity. Additionally, the performance is evaluated primarily on LibriSpeech, which consists of read speech; performance on conversational or noisy speech is not reported. The "one-step" nature also implies that the model cannot easily refine outputs iteratively, which might be a limitation for applications requiring high-fidelity correction of errors. The paper notes that larger models did not improve performance, suggesting that the one-step constraint is difficult to scale without further architectural or training innovations.
Broader impact
This work has significant implications for the deployment of LLM-based TTS systems, particularly in latency-sensitive applications like real-time voice assistants, gaming, and on-device AI. By enabling high-quality, one-step audio generation, it reduces computational costs and energy consumption, making advanced speech synthesis more accessible. The approach could also be extended to other modalities where low-latency generation is critical. However, as with all audio generation technologies, there are potential misuse cases regarding voice cloning and deepfakes, necessitating responsible deployment practices. This paper presents a significant advancement in low-latency neural audio synthesis by successfully adapting MeanFlow to a latent space for one-step Token-to-Waveform generation, achieving a substantial reduction in inference latency while maintaining high perceptual quality through innovative refinement strategies.
Author of this paper? Send a correction.