VoXtream2: Full-stream TTS with dynamic speaking rate control
Abstract
Full-stream text-to-speech (TTS) for interactive systems must start speaking with minimal delay while remaining controllable as text arrives incrementally. We present VoXtream2, a zero-shot full-stream TTS model with dynamic speaking-rate control that can be updated mid-utterance on the fly. VoXtream2 combines a distribution matching mechanism over duration states with classifier-free guidance across conditioning signals to improve controllability and synthesis quality. Prompt-text masking enables textless audio prompting, removing the need for prompt transcription. Across standard zero-shot benchmarks and a dedicated speaking-rate test set, VoXtream2 achieves competitive objective and subjective results against public baselines despite a smaller model and less training data. In full-stream mode, it runs 4 times faster than real time with 74 ms first-packet latency on a consumer GPU.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
VoXtream2 presents a significant advancement in full-stream TTS systems by introducing dynamic speaking rate control and improving synthesis quality through innovative methodologies. The combination of low latency, high intelligibility, and adaptability positions this work as a valuable contribution to the field of machine learning and audio synthesis, with potential applications in various interactive systems.
Methodology
The methodology presented in VoXtream2 is innovative, particularly in its approach to dynamic speaking rate control and the integration of classifier-free guidance (CFG) for improved speech synthesis. The use of distribution matching over duration states and prompt text masking to enable textless audio prompting demonstrates a significant advancement in TTS systems, addressing key limitations of previous models. The architecture builds on the earlier VoXtream model but introduces critical enhancements that allow for real-time, low-latency speech generation while maintaining high intelligibility and voice cloning capabilities. The detailed description of the model architecture, including the use of the International Phonetic Alphabet (IPA) and the autoregressive Temporal Transformer (TT), illustrates a thoughtful design process aimed at achieving both speed and quality.
Results & evaluation
The experimental evaluation is robust, utilizing a variety of datasets, including the Emilia spontaneous speech dataset and the HiFiTTS-2 dataset. The results are compared against several state-of-the-art models, showcasing competitive performance in both objective and subjective metrics. The paper includes comprehensive evaluations of static and dynamic speaking rate control, with clear metrics for intelligibility, speaker similarity, and naturalness. The use of both human evaluations and objective metrics like WER, SPK-SIM, and UTMOS adds credibility to the findings. However, the reliance on specific datasets may limit the generalizability of the results.
Reproducibility
The paper provides sufficient detail regarding the model architecture, training procedures, and evaluation metrics, which supports reproducibility. However, the absence of a publicly available code repository limits the ease with which other researchers can replicate the results. The authors mention using a specific implementation of the Llama-3.2 transformer, which is beneficial, but further details on hyperparameters and training conditions would enhance reproducibility.
Limitations
The paper acknowledges several limitations, including the influence of the acoustic prompt's speaking rate on the generated speech rate and the complexity of the data preprocessing pipeline. The model's performance may degrade under certain conditions, particularly when generating speech at extreme speaking rates. Additionally, while the dynamic speaking rate control is a significant advancement, the model still exhibits some dependency on the prompt rate, which could be a barrier to achieving fully independent control.
Broader impact
VoXtream2 has the potential to significantly impact the development of conversational agents and real-time TTS applications, particularly in scenarios requiring low latency and high adaptability. The ability to generate speech that closely mimics human-like dynamics in speaking rate could enhance user experience in voice-driven interfaces, making interactions more natural and engaging. Furthermore, the advancements in voice cloning and intelligibility could have applications in accessibility technologies and personalized voice synthesis. VoXtream2 presents a significant advancement in full-stream TTS systems by introducing dynamic speaking rate control and improving synthesis quality through innovative methodologies. The combination of low latency, high intelligibility, and adaptability positions this work as a valuable contribution to the field of machine learning and audio synthesis, with potential applications in various interactive systems.
Author of this paper? Send a correction.