VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
Abstract
Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in. Recent duplex speech-to-speech and speech-to-text models reduce latency by replacing multi-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized jointly. We propose VoiceChat-TTS, a low-latency, continuous, and streamable text-to-speech model for interactive agents. VoiceChat-TTS is driven directly by LLM text-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available. The model enables always-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid-utterance interruptions without resetting the KV cache.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
VoiceChat-TTS presents a robust, low-latency, and interruptible streaming TTS system that effectively bridges the gap between high-quality offline synthesis and the real-time demands of full-duplex interactive agents. The comprehensive evaluation, including detailed latency analysis and interruption handling benchmarks, demonstrates its practical utility and sets a strong baseline for future research in continuous speech synthesis.
Methodology
The paper proposes VoiceChat-TTS, a continuous, streamable TTS model built upon the Audio Flamingo 3-Chat architecture. The core technical contributions involve adapting a streaming decoder for full-duplex interaction. Key modifications include: 1) A Character-Aware Subword Encoder to handle out-of-vocabulary subwords from LLMs by converting them to character sequences processed by a shallow Transformer. 2) An interruption mechanism using explicit control tokens to halt generation and transition to silence without resetting the KV cache. 3) Audio Prompt Conditioning using a 3-second reference audio to stabilize speaker identity at the start of generation, addressing the lack of context in streaming settings. 4) A Mixture of Gaussian Estimation Head (MoGH) for faster RVQ token decoding. The approach is pragmatic, focusing on engineering solutions (gated fusion, specific tokenization, silence modeling) to enable real-time, interruptible speech synthesis. While not fundamentally novel in terms of architecture (it extends existing streaming decoders), the specific integration of these components for a unified duplex TTS system is a valuable engineering contribution.
Results & evaluation
The evaluation is comprehensive and well-structured. The authors compare VoiceChat-TTS against strong offline (Chatterbox-TTS, Qwen3-TTS) and streaming (Audio Flamingo 3-Chat) baselines. Metrics include CER, WER, SECS (speaker similarity), and Squim-MOS. The results show that VoiceChat-TTS achieves competitive quality, significantly outperforming the base Audio Flamingo 3-Chat decoder in intelligibility and quality. The paper also includes a dedicated interruption evaluation using the Full-Duplex-Bench (FDB) subset, measuring Stop Latency and Leakage. The latency analysis is particularly strong, providing detailed breakdowns of acoustic-token ITL and codec decoding time on specific hardware (RTX A6000), demonstrating a 2.1x speedup over a comparable baseline. The use of both synthetic and real conversational data for training is well-justified.
Reproducibility
The paper provides significant detail for reproduction. It specifies the model size (977M parameters), the base architecture (Gemma 3-based), the codec configuration (12.5 Hz, 31-codebook RVQ), and the training stages (pretraining on single-turn data, fine-tuning on multi-turn). The code is publicly available via NVIDIA NeMo Speech, and the model checkpoint is on Hugging Face. The training data sources are listed (LibriTTS, HiFiTTS, synthetic data, Fisher). The evaluation protocol is also clearly defined, including the use of Silero VAD for interruption timing detection. This high level of detail ensures that the work is reproducible.
Limitations
The authors acknowledge several limitations. First, the model lacks user-audio conditioning, meaning it cannot dynamically adapt its prosody to the user's speech characteristics or interruptions in real-time (it relies on text tokens). Second, there is a lack of controlled component-wise ablations; while preliminary experiments suggested cumulative gains, a systematic ablation study is missing. Third, speaker similarity (SECS) degrades over longer sequences of continuous generation, particularly for unseen speakers, indicating a challenge in maintaining long-context speaker consistency. Finally, the evaluation of "silence" generation relies on ASR-based metrics, which may not fully capture the perceptual quality of silence or the naturalness of the transition into/out of silence.
Broader impact
VoiceChat-TTS contributes to the development of more natural and responsive human-computer interaction systems. By enabling low-latency, interruptible speech synthesis, it facilitates the deployment of always-on voice assistants and conversational agents that behave more like human interlocutors. This has positive implications for accessibility, customer service, and companion AI. However, the ease of generating high-quality, realistic speech also raises concerns about potential misuse in deepfakes or deceptive interactions, although the model's reliance on text input mitigates some of the risks associated with direct audio-to-audio generation. VoiceChat-TTS presents a robust, low-latency, and interruptible streaming TTS system that effectively bridges the gap between high-quality offline synthesis and the real-time demands of full-duplex interactive agents. The comprehensive evaluation, including detailed latency analysis and interruption handling benchmarks, demonstrates its practical utility and sets a strong baseline for future research in continuous speech synthesis.
Author of this paper? Send a correction.