VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency
Abstract
We present VoXtream, a fully autoregressive, zero-shot streaming text-to-speech (TTS) system for real-time use that begins speaking from the first word. VoXtream directly maps incoming phonemes to audio tokens using a monotonic alignment scheme and a dynamic look-ahead that does not delay onset. Built around an incremental phoneme transformer, a temporal transformer predicting semantic and duration tokens, and a depth transformer producing acoustic tokens, VoXtream achieves, to our knowledge, the lowest initial delay among publicly available streaming TTS: 102 ms on GPU. Despite being trained on a mid-scale 9k-hour corpus, it matches or surpasses larger baselines on several metrics, while delivering competitive quality in both output- and full-streaming settings. Demo and code are available at https://herimor.github.io/voxtream.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
VoXtream presents a pioneering approach to streaming TTS with ultra-low latency, combining innovative transformer architectures to achieve competitive performance. The paper's contributions are substantial, addressing a critical need in real-time speech synthesis and setting a new benchmark for future research in the field.
Methodology
The methodology presented in VoXtream is innovative, utilizing a combination of autoregressive transformers to achieve low-latency streaming TTS. The architecture's design, which includes an incremental Phoneme Transformer, a Temporal Transformer, and a Depth Transformer, is well thought out and addresses the critical issue of initial latency in TTS systems. The use of dynamic look-ahead for phoneme processing is particularly noteworthy, as it allows for immediate speech output without waiting for the entire input, which is a significant advancement over existing models. The integration of these components into a cohesive framework demonstrates a solid understanding of the challenges in TTS and offers a practical solution.
Results & evaluation
The experimental evaluation is robust, with comprehensive testing on established datasets such as SEED-TTS and LibriSpeech. The paper provides clear comparisons with multiple baseline models, showcasing VoXtream's performance in terms of intelligibility, naturalness, and latency. The results indicate that VoXtream not only meets but often exceeds the performance of larger models, despite being trained on a smaller dataset. The use of both objective metrics (WER, SPK-SIM, UTMOS) and subjective evaluations through user studies strengthens the credibility of the findings.
Reproducibility
The paper includes sufficient implementation details, such as model architecture specifications, training procedures, and evaluation metrics, which facilitate reproducibility. However, the absence of a publicly accessible code repository limits the ease with which other researchers can replicate the results. The authors mention the use of specific datasets and training setups, which is helpful, but a direct link to the code would enhance reproducibility further.
Limitations
One limitation of the study is the reliance on a mid-scale dataset (9k hours), which may restrict the model's generalizability compared to systems trained on larger datasets. Additionally, while the model achieves low initial latency, the paper does not extensively discuss the trade-offs in quality that may arise from such optimizations. The subjective evaluations, while positive, could benefit from a larger participant pool to ensure broader applicability of the results.
Broader impact
The implications of VoXtream are significant for real-time applications in conversational AI, voice assistants, and simultaneous translation systems. The ability to generate speech with minimal latency enhances user experience and engagement, making it a valuable contribution to the field of speech synthesis. The model's architecture could inspire further research into low-latency systems and their applications in various domains, potentially leading to advancements in human-computer interaction. VoXtream presents a pioneering approach to streaming TTS with ultra-low latency, combining innovative transformer architectures to achieve competitive performance. The paper's contributions are substantial, addressing a critical need in real-time speech synthesis and setting a new benchmark for future research in the field.
Author of this paper? Send a correction.