ARCHI-TTS: A flow-matching-based Text-to-Speech Model with Self-supervised Semantic Aligner and Accelerated Inference
Abstract
Although diffusion-based, non-autoregressive text-to-speech (TTS) systems have demonstrated impressive zero-shot synthesis capabilities, their efficacy is still hindered by two key challenges: the difficulty of text-speech alignment modeling and the high computational overhead of the iterative denoising process. To address these limitations, we propose ARCHI-TTS that features a dedicated semantic aligner to ensure robust temporal and semantic consistency between text and audio. To overcome high computational inference costs, ARCHI-TTS employs an efficient inference strategy that reuses encoder features across denoising steps, drastically accelerating synthesis without performance degradation. An auxiliary CTC loss applied to the condition encoder further enhances the semantic understanding. Experimental results demonstrate that ARCHI-TTS achieves a WER of 1.98% on LibriSpeech-PC test-clean, and 1.47%/1.42% on SeedTTS test-en/test-zh with a high inference efficiency, consistently outperforming recent state-of-the-art TTS systems.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of ARCHI-TTS, a novel non-autoregressive text-to-speech model that effectively addresses the challenges of text-speech alignment and computational efficiency through innovative architectural components. The comprehensive analysis of its technical contributions, methodology, and results positions it as a significant advancement in the TTS domain, with potential for impactful applications in various audio synthesis tasks.
Methodology
The methodology proposed in ARCHI-TTS is innovative, combining a semantic aligner with a flow-matching decoder to address the challenges of text-speech alignment and inference efficiency in TTS systems. The use of a low-token-rate representation derived from a Variational Autoencoder (VAE) is a significant advancement, allowing for a more compact representation of audio data while maintaining quality. The architecture's reliance on a transformer-based semantic aligner to create self-supervised text-aligned semantic representations is a novel approach that enhances the model's ability to generate coherent and contextually relevant speech. The integration of an auxiliary CTC loss to bolster semantic understanding further demonstrates a thoughtful approach to improving the model's performance.
Results & evaluation
The experimental evaluation is robust, utilizing a large-scale multilingual dataset (100k hours) for training and multiple established benchmarks for testing. The reported results, including a WER of 1.98% on the LibriSpeech-PC test-clean and competitive performance on the SeedTTS test set, indicate that ARCHI-TTS outperforms several state-of-the-art models while using fewer computational resources. The inclusion of ablation studies adds depth to the evaluation, providing insights into the contributions of various architectural components. However, the paper could benefit from more extensive subjective evaluations to further validate the quality of the generated speech.
Reproducibility
The paper provides sufficient details regarding the model configuration, training process, and evaluation metrics, which should facilitate reproducibility. The authors mention the use of specific hardware (8 RTX 5090 GPUs) and training duration, which are valuable for replicating the experiments. However, the lack of a direct link to the code repository limits accessibility for other researchers wishing to reproduce the results.
Limitations
While the proposed model shows promising results, it does exhibit some limitations, such as slightly lagging behind other state-of-the-art models in subjective quality evaluations. The reliance on a specific dataset (Emilia) may also limit the generalizability of the findings. Additionally, the computational efficiency improvements come at the cost of some performance degradation, which may need further exploration.
Broader impact
The advancements presented in ARCHI-TTS have significant implications for the field of TTS and audio synthesis, particularly in enhancing the efficiency and quality of speech generation. The model's ability to perform zero-shot synthesis with high fidelity could lead to broader applications in voice cloning, audiobooks, and interactive voice response systems. As TTS technology continues to evolve, the methodologies introduced in this paper could influence future research directions and commercial applications. The main contribution of this paper is the introduction of ARCHI-TTS, a novel non-autoregressive text-to-speech model that effectively addresses the challenges of text-speech alignment and computational efficiency through innovative architectural components. The comprehensive analysis of its technical contributions, methodology, and results positions it as a significant advancement in the TTS domain, with potential for impactful applications in various audio synthesis tasks.
Author of this paper? Send a correction.