U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation
Abstract
We propose \textbf{U-Codec}, an \textbf{U}ltra low frame-rate neural speech \textbf{Codec} that achieves high-fidelity reconstruction and fast speech generation at an extremely low frame-rate of 5Hz (5 frames per second). Extreme compression at 5Hz typically leads to severe intelligibility and spectral detail loss, we introduce a Transformer-based inter-frame long-term dependency module and systematically explore residual vector quantization (RVQ) depth and codebook size to identify optimal configurations. Moreover, we apply U-Codec into a large language model (LLM)-based auto-regressive TTS model, which leverages global and local hierarchical architecture to effectively capture dependencies across multi-layer tokens. We extend LLM-based TTS from 3-layer RVQ at 50Hz to 32-layer RVQ at 5Hz. Experimental results demonstrate that U-Codec improves LLM-based TTS inference speed by around 3 $\times$ over high-frame-rate codecs while maintaining similarity and naturalness. These results validate the feasibility of using highly compressed 5Hz discrete tokens for fast and high-fidelity speech synthesis.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The U-Codec presents a significant advancement in neural speech codecs, achieving high-fidelity speech synthesis at an unprecedented low frame rate of 5Hz, thereby enhancing computational efficiency while maintaining quality. The innovative methodology and comprehensive experimental validation position this work as a notable contribution to the field of machine learning and audio processing.
Methodology
The proposed U-Codec introduces a novel architecture that combines a Transformer-based inter-frame long-term dependency module with a hierarchical global-local Transformer architecture, effectively addressing the challenges of speech synthesis at ultra-low frame rates. The systematic exploration of residual vector quantization (RVQ) depth and codebook size is well-structured, providing a comprehensive approach to optimizing speech quality under extreme compression. The introduction of the Codecformer network is particularly innovative, as it allows for efficient modeling of long sequences while maintaining high fidelity.
Results & evaluation
The experiments are robust, utilizing a large-scale multilingual dataset and a variety of evaluation metrics such as WER, PESQ, and STOI to assess performance. The results demonstrate significant improvements in inference speed and speech quality compared to existing high-frame-rate codecs, establishing a new benchmark in the field. However, the paper could benefit from additional comparative analyses with more recent state-of-the-art methods.
Reproducibility
The paper provides a clear description of the training setup, datasets, and evaluation metrics, which enhances reproducibility. The release of a demo and code is a positive aspect, allowing others to validate the findings. However, specific implementation details, such as hyperparameters and training configurations, could be more thoroughly documented.
Limitations
While the U-Codec shows promising results, it does not yet match the performance of certain high-frame-rate systems in terms of PESQ and SPK-SIM. Additionally, the complexity of the model increases with deeper RVQ stacks, which may limit practical applications in resource-constrained environments.
Broader impact
The U-Codec has the potential to significantly impact real-time speech synthesis applications, especially in scenarios where computational efficiency is critical, such as mobile devices and low-latency communication systems. Its ability to maintain high fidelity at ultra-low frame rates could lead to advancements in various fields, including virtual assistants, gaming, and accessibility tools. The U-Codec presents a significant advancement in neural speech codecs, achieving high-fidelity speech synthesis at an unprecedented low frame rate of 5Hz, thereby enhancing computational efficiency while maintaining quality. The innovative methodology and comprehensive experimental validation position this work as a notable contribution to the field of machine learning and audio processing.
Author of this paper? Send a correction.