StellarTTS: Sparse Temporal Embedding for Low-Latency and Robust Speech Synthesis
Abstract
The trade-off between robustness, latency, and prosody critically challenges text-to-speech (TTS) systems. Autoregressive models, despite fidelity, are slow and error-prone; non-autoregressive (NAR) alternatives, while fast, often sacrifice prosodic naturalness via rigid alignments. This paper introduces StellarTTS, a novel mobile-optimized NAR TTS framework based on a sparse temporal embedding strategy, enabling granular control of phoneme duration, pronunciation, and prosody. Furthermore, we propose a semantic-aware codec that facilitates efficient single-stage decoding. Conditioned on the sparse temporal embedding, our 83M-parameter lightweight masked generative transformer achieves a real-time factor (RTF) of 0.08. Experiments demonstrate that StellarTTS attains lower latency and stronger robustness compared to state-of-the-art TTS systems, while maintaining competitive performance in audio quality, prosodic naturalness, and speaker similarity.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
StellarTTS presents a practical, mobile-optimized NAR TTS system that prioritizes robustness and latency through a novel sparse temporal embedding and semantic-aware codec, achieving significant efficiency gains at the cost of some speaker fidelity.
Methodology
The paper proposes StellarTTS, a non-autoregressive (NAR) Text-to-Speech (TTS) framework optimized for mobile deployment. The core methodological contributions are twofold: (1) a "semantic-aware codec" that uses Residual Vector Quantization (RVQ) with the 0-th channel distilled from Wav2vec-Bert semantic features, aiming to unify semantic and acoustic token prediction into a single stage; and (2) "sparse temporal embedding," which replaces rigid length regulators with a strategy where phone embeddings are placed only at the center of the duration window, allowing the model to learn prosodic variations while maintaining structural anchors. The architecture is an 83M-parameter masked generative transformer. While the integration of semantic distillation into the codec is a practical engineering choice for latency reduction, the "sparse temporal embedding" is a relatively minor variation of existing alignment-free or soft-alignment techniques in masked generative models (e.g., MaskGCT, F5-TTS). The novelty lies more in the specific system integration for mobile constraints than in fundamental algorithmic innovation.
Results & evaluation
The experimental evaluation is comprehensive in terms of metrics but reveals a significant trade-off. The model achieves a Real-Time Factor (RTF) of 0.08, demonstrating high efficiency. It shows superior Word Error Rate (WER) on challenging datasets (test-hard), indicating robustness against error propagation common in AR models. However, the speaker similarity (SIM-o) is notably lower than state-of-the-art baselines (e.g., MaskGCT, CosyVoice2). The authors attribute this to the semantic distillation in the codec, which compresses acoustic detail. Subjective evaluations (CMOS/SMOS) show competitive naturalness but do not fully compensate for the objective similarity gap. The ablation studies support the efficacy of the sparse embedding strategy over random anchoring. The comparison with baselines is fair, but the lower SIM-o is a critical limitation for zero-shot TTS applications where speaker fidelity is paramount.
Reproducibility
The paper provides sufficient architectural details, including the RVQ structure, loss functions (KL divergence for semantic distillation, MSE for duration prediction, cross-entropy for generation), and training hyperparameters (AdamW, learning rate schedule). The dataset (Emilia) and test sets (Seed-TTS) are standard. However, the specific implementation details of the "semantic-aware codec" training joint optimization and the exact configuration of the duration predictor are somewhat high-level. The code link provided is a demo page, not a GitHub repository, which hinders immediate reproducibility.
Limitations
The primary limitation is the reduced speaker similarity (SIM-o) compared to SOTA models, which is a direct consequence of the single-stage semantic-aware codec design. The model's performance on very long-form synthesis is not explicitly evaluated, though the NAR nature suggests potential for consistency issues over long contexts. The reliance on a pre-trained semantic encoder (Wav2vec-Bert) for distillation adds an external dependency, though it is frozen during the main generation phase. The "sparse" embedding strategy, while effective for robustness, may struggle with extremely rapid prosodic changes that do not align well with the phone-centric anchor points.
Broader impact
This work contributes to the democratization of high-quality TTS on edge devices, which has significant implications for privacy-preserving voice assistants and low-latency communication tools. By demonstrating that robust, low-latency TTS is possible without sacrificing too much naturalness, it pushes the field towards more practical, deployable AI audio systems. However, the lower speaker similarity might limit its use in high-fidelity cloning applications. StellarTTS presents a practical, mobile-optimized NAR TTS system that prioritizes robustness and latency through a novel sparse temporal embedding and semantic-aware codec, achieving significant efficiency gains at the cost of some speaker fidelity.
Author of this paper? Send a correction.