CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment
Abstract
Large-language-model (LLM)-based text-to-speech (TTS) systems can generate natural speech, but most are not designed for low-latency dual-streaming synthesis. High-quality dual-streaming TTS depends on accurate text--speech alignment and well-designed training sequences that balance synthesis quality and latency. Prior work often relies on GMM-HMM based forced-alignment toolkits (e.g., MFA), which are pipeline-heavy and less flexible than neural aligners; fixed-ratio interleaving of text and speech tokens struggles to capture text--speech alignment regularities. We propose CTC-TTS, which replaces MFA with a CTC based aligner and introduces a bi-word based interleaving strategy. Two variants are designed: CTC-TTS-L (token concatenation along the sequence length) for higher quality and CTC-TTS-F (embedding stacking along the feature dimension) for lower latency. Experiments show that CTC-TTS outperforms fixed-ratio interleaving and MFA-based baselines on streaming synthesis and zero-shot tasks. Speech samples are available at https://ctctts.github.io/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents CTC-TTS, a novel dual-streaming text-to-speech synthesis method that leverages CTC alignment and bi-word interleaving strategies. The approach demonstrates significant improvements in synthesis quality and latency, marking a meaningful contribution to the field of audio processing and machine learning.
Methodology
The paper introduces a novel approach to text-to-speech synthesis by employing a Connectionist Temporal Classification (CTC)-based alignment mechanism, which is a significant departure from traditional GMM-HMM forced alignment methods. The introduction of a bi-word interleaving strategy enhances the model's ability to capture temporal dependencies between text and speech, addressing the limitations of fixed-ratio interleaving. The two variants, CTC-TTS-L and CTC-TTS-F, are well-defined and cater to different quality-latency trade-offs, showcasing a thoughtful design that balances synthesis quality with operational efficiency. The methodology is sound, with clear explanations of the alignment process and interleaving strategies.
Results & evaluation
The experimental setup is robust, utilizing well-known datasets such as LibriSpeech and VoiceAssistant400K for evaluation. The authors provide comprehensive results comparing their method against established baselines, demonstrating significant improvements in both streaming synthesis and zero-shot tasks. The use of objective metrics like Word Error Rate (WER) and Character Error Rate (CER) alongside subjective evaluations (Mean Opinion Score) adds credibility to the findings. The results indicate that both CTC-TTS variants outperform existing methods, validating the proposed approach.
Reproducibility
The paper includes detailed implementation details, including model architecture, training configurations, and evaluation metrics, which enhance reproducibility. However, the absence of a publicly available code repository at this stage may hinder full reproducibility until the code is released post-acceptance.
Limitations
While the proposed method shows promising results, the paper does not address potential limitations regarding the scalability of the approach to multi-speaker scenarios or its performance in diverse linguistic contexts. Additionally, the reliance on a specific G2P model (Phonetisaurus) may limit flexibility in phoneme generation.
Broader impact
The advancements in low-latency, high-quality TTS systems have significant implications for applications in virtual assistants, audiobooks, and real-time communication tools. The ability to synthesize speech with reduced latency while maintaining naturalness can enhance user experience in various interactive applications. The research could pave the way for further innovations in speech synthesis technologies. The paper presents CTC-TTS, a novel dual-streaming text-to-speech synthesis method that leverages CTC alignment and bi-word interleaving strategies. The approach demonstrates significant improvements in synthesis quality and latency, marking a meaningful contribution to the field of audio processing and machine learning.
Author of this paper? Send a correction.