MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control
Abstract
MambaVoiceCloning (MVC) asks whether the conditioning path of diffusion-based TTS can be made fully SSM-only at inference, removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody, while preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time O(T) conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba-TTS systems that remain hybrid at inference, MVC removes attention-based duration and style modules under a fixed StyleTTS2 mel-diffusion-vocoder backbone. Trained on LJSpeech/LibriTTS and evaluated on VCTK, CSS10 (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but statistically reliable gains over StyleTTS2, VITS, and Mamba-attention hybrids in MOS/CMOS, F0 RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by 1.6x. Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and deployability.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MambaVoiceCloning, a novel TTS system that utilizes a fully state-space model conditioning path to improve efficiency and stability while eliminating traditional attention mechanisms. This work significantly advances the field of TTS by addressing key limitations in existing architectures, offering a promising direction for future research and applications.
Methodology
The methodology of MambaVoiceCloning (MVC) is innovative in its approach to eliminate attention and recurrent layers in a text-to-speech (TTS) system, utilizing a fully state-space model (SSM) conditioning path. The integration of gated bidirectional Mamba text encoders, Temporal Bi-Mamba, and Expressive Mamba with AdaLN modulation showcases a significant architectural redesign that emphasizes efficiency and stability. The linear-time complexity and bounded activation memory are particularly noteworthy, as they address common issues in traditional TTS systems, such as memory pressure and drift in long sequences. The paper provides a clear explanation of the architecture and its components, supported by rigorous theoretical grounding.
Results & evaluation
The experimental evaluation is comprehensive, utilizing multiple datasets including LJSpeech, LibriTTS, VCTK, and CSS10, which allows for a robust assessment of MVC's performance across various conditions. The paper reports both subjective (MOS, CMOS) and objective metrics (F0 RMSE, MCD, WER), demonstrating statistically significant improvements over baseline models. The inclusion of long-form and cross-lingual evaluations further strengthens the findings, showcasing the model's generalization capabilities. However, while the improvements are statistically reliable, they are described as modest, indicating room for further enhancement.
Reproducibility
The authors provide detailed implementation and training protocols, ensuring that the methodology can be reproduced. The use of a unified optimization schedule across all models and the provision of code on GitHub enhances reproducibility. However, the paper could benefit from more explicit details regarding hyperparameter tuning and the specific configurations used for each model.
Limitations
The paper acknowledges limitations such as the focus on conditioning efficiency over fine-grained emotion control, and the model's training solely on English datasets, which may affect its performance in multilingual contexts. Additionally, the diffusion decoder remains the primary latency bottleneck, which could hinder real-time applications.
Broader impact
The MVC framework has potential implications for real-time TTS applications, particularly in scenarios requiring efficient memory usage and low latency. Its architecture could serve as a drop-in replacement for existing TTS systems, enhancing their deployability in resource-constrained environments. The focus on ethical considerations, such as watermarking and speaker consent, is commendable and highlights the responsible deployment of AI technologies. The main contribution of this paper is the introduction of MambaVoiceCloning, a novel TTS system that utilizes a fully state-space model conditioning path to improve efficiency and stability while eliminating traditional attention mechanisms. This work significantly advances the field of TTS by addressing key limitations in existing architectures, offering a promising direction for future research and applications.
Author of this paper? Send a correction.