SemaVoice: Semantic-Aware Continuous Autoregressive Speech Synthesis
Abstract
Continuous autoregressive speech synthesis has recently emerged as a promising direction for zero-shot text-to-speech (TTS). However, existing methods still suffer from a fundamental mismatch between semantic-prosodic modeling and reconstruction-driven continuous speech representations. This mismatch causes TTS models to focus excessively on low-level acoustic textures at the expense of high-level semantic coherence, further exacerbating error accumulation in autoregressive generation. To address this challenge, we propose SemaVoice, a semantic-aware continuous autoregressive framework for high-fidelity zero-shot TTS. SemaVoice introduces a Speech Foundation Model (SFM) guided alignment mechanism that refines continuous speech representations to better capture both local semantic consistency and global structural relationships. These representations condition a patch-wise diffusion head within the autoregressive framework for high-quality speech synthesis. Experimental results on the Seed-TTS benchmark show that SemaVoice achieves an English WER of 1.71\% and remains highly competitive with state-of-the-art open-source systems in both objective and subjective evaluations. The effectiveness of SFM guided alignment is further confirmed by significant improvements under varying representation granularities with a fixed information-rate constraint.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SemaVoice, a semantic-aware continuous autoregressive framework that significantly improves high-fidelity zero-shot text-to-speech synthesis through an innovative SFM-guided alignment mechanism. This work represents a meaningful advancement in the field of speech synthesis, addressing critical limitations in existing models and demonstrating strong experimental results.
Methodology
The proposed SemaVoice framework introduces a novel SFM-guided alignment mechanism that effectively addresses the mismatch between semantic-prosodic modeling and reconstruction-driven continuous speech representations. This innovative approach enhances the semantic coherence of generated speech while maintaining acoustic fidelity. The use of a continuous autoregressive framework, combined with a patch-wise diffusion head, is a significant advancement over traditional TTS architectures. The methodology is well-structured, with a clear explanation of the components and their interactions, although the complexity may pose challenges for replication.
Results & evaluation
The experimental evaluation is robust, utilizing a large-scale bilingual dataset of 150K hours for training and thorough testing on the Seed-TTS benchmark. The results demonstrate competitive performance against state-of-the-art systems in both objective (WER, speaker similarity) and subjective (MOS) metrics, indicating the effectiveness of the proposed framework. The ablation studies provide valuable insights into the contributions of key components, reinforcing the importance of the SFM-guided alignment mechanism.
Reproducibility
While the paper provides a detailed description of the architecture and training process, it lacks a publicly available implementation or code repository, which hinders reproducibility. The absence of a demo URL also limits practical engagement with the model.
Limitations
The evaluation is limited to a bilingual dataset, which may restrict the generalizability of the findings. Additionally, the paper acknowledges inherent challenges with sequential inference latency and error accumulation in autoregressive generation, which could impact real-time applications.
Broader impact
The advancements in zero-shot TTS synthesis have significant implications for applications in voice cloning, virtual assistants, and accessibility technologies. By improving the semantic coherence and acoustic fidelity of synthesized speech, SemaVoice could enhance user experience in various domains, including entertainment, education, and communication. The main contribution of this paper is the introduction of SemaVoice, a semantic-aware continuous autoregressive framework that significantly improves high-fidelity zero-shot text-to-speech synthesis through an innovative SFM-guided alignment mechanism. This work represents a meaningful advancement in the field of speech synthesis, addressing critical limitations in existing models and demonstrating strong experimental results.
Author of this paper? Send a correction.