SemBridge: Semantic Token Anchoring for Continuous-Latent Autoregressive Speech Generation
Abstract
Continuous-latent autoregressive speech generation has emerged as a promising alternative to discrete-token modeling by avoiding quantization loss and preserving richer acoustic information. However, continuous acoustic targets do not ex- pose linguistic structure as explicit token-level prediction tar- gets. Consequently, the autoregressive language model (LM) must acquire linguistic structure indirectly through acous- tic prediction, which can compromise the content fidelity of generated speech. We propose SemBridge, a training-only semantic-token anchoring framework for continuous-latent autoregressive speech generation. SemBridge uses discrete se- mantic tokens to directly supervise autoregressive LM states and employs a Semantic-Aligned Acoustic VAE to organize the continuous target space under the same semantic refer- ence. The semantic supervision is used only during train- ing, while inference remains entirely continuous. We evalu- ate SemBridge on zero-shot text-to-speech (TTS) and score- conditioned singing voice synthesis (SVS). Across multi- ple benchmarks, SemBridge improves content accuracy, as measured by word and character error rates (WER/CER), while maintaining competitive speaker similarity and percep- tual quality. Experimental results demonstrate that explicit semantic-token supervision for autoregressive state learning is an effective and general direction for continuous speech generation. Speech samples are available.1 The model code and checkpoints will be available at https://github.com/ASLP- lab/SemBridge
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SemBridge introduces a training-only semantic-token anchoring framework that improves content fidelity in continuous-latent autoregressive speech generation by supervising LM states with discrete semantic tokens, demonstrating significant gains in WER/CER across TTS and SVS benchmarks while maintaining competitive acoustic quality.
Methodology
The paper proposes SemBridge, a framework for continuous-latent autoregressive speech generation. The core innovation is "semantic-token anchoring," which uses discrete semantic tokens (from a frozen GLM-4 tokenizer) to supervise the hidden states of the autoregressive language model during training. This is complemented by a Semantic-Aligned Acoustic VAE (SA-VAE) that aligns continuous acoustic latents with semantic embeddings. The approach is technically sound and addresses a real gap in continuous speech generation: the lack of explicit linguistic structure in continuous targets. However, the novelty is moderate; using discrete tokens to guide continuous generation is a known strategy (e.g., in VALL-E X, SpeechTokenizer), and the specific implementation of anchoring LM states with classification loss is a straightforward adaptation of existing multi-task learning principles. The method is essentially a training-time regularization technique that does not alter the inference architecture, which is a pragmatic but not groundbreaking design choice.
Results & evaluation
The evaluation is comprehensive, covering zero-shot TTS on Seed-TTS-Eval and CV3-Eval, as well as score-conditioned SVS on GMO-SVS. The results show consistent improvements in content accuracy (WER/CER) compared to baselines like MELA-TTS and VoxCPM, while maintaining competitive speaker similarity and perceptual quality. The ablation studies are well-conducted, isolating the effects of target-space alignment, state anchoring, anchoring depth, and anchoring strength. The inclusion of SVS results adds value by demonstrating the method's transferability to a more challenging task with explicit musical constraints. The comparison with a wide range of baselines (discrete, continuous, and hybrid) provides a strong context for the results. The paper effectively demonstrates that explicit semantic supervision improves content fidelity without sacrificing acoustic quality.
Reproducibility
The paper provides detailed descriptions of the model architecture, training stages, hyperparameters, and datasets. It specifies the use of the GLM-4 tokenizer and the SA-VAE architecture. The training configurations, including batch sizes, learning rates, and loss weights, are clearly listed. The authors state that code and checkpoints will be available, and a demo link is provided. The detailed ablation studies and clear separation of Stage I and Stage II training procedures enhance reproducibility. However, the reliance on specific internal datasets (Soul AI Lab's 20K-hour singing data) for the joint TTS-SVS model limits full reproducibility for that specific configuration, though the TTS-only experiments are likely reproducible with the public VoxBox corpus.
Limitations
The primary limitation is that the semantic supervision is only applied during training. While this keeps inference efficient, it means the model relies entirely on the learned alignment between continuous latents and discrete tokens, which might not be as robust as end-to-end discrete modeling in some edge cases. The paper notes a trade-off between content accuracy and synthesis quality depending on the anchoring depth and weight, suggesting that hyperparameter tuning is critical. Additionally, the use of a frozen semantic tokenizer means the model cannot adapt the semantic representation to the specific domain of speech generation, potentially missing out on speech-specific semantic nuances that could be learned jointly. The improvement in WER/CER, while statistically significant, is not massive, and the gains in perceptual quality (UTMOS) are marginal.
Broader impact
SemBridge contributes to the field of speech synthesis by providing a robust method for improving content fidelity in continuous-latent models. This can lead to more reliable and accurate text-to-speech and singing voice synthesis systems, which have broad applications in accessibility, entertainment, and human-computer interaction. The method's ability to improve lyric intelligibility in SVS is particularly relevant for the music industry. However, the potential for misuse in generating deepfakes or misleading audio content remains a concern, as with any advanced speech synthesis technology. The paper does not explicitly discuss safety mitigations, which is a common omission in technical research but important for broader impact assessment. SemBridge introduces a training-only semantic-token anchoring framework that improves content fidelity in continuous-latent autoregressive speech generation by supervising LM states with discrete semantic tokens, demonstrating significant gains in WER/CER across TTS and SVS benchmarks while maintaining competitive acoustic quality.
Author of this paper? Send a correction.