SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization
Abstract
Speech codecs that convert continuous speech signals into discrete tokens have become essential for speech language models (SLMs). However, existing codecs struggle to balance high-quality reconstruction with semantically rich representations, limiting their effectiveness in both generative and understanding tasks. In this work, we propose SAC, a neural speech codec with semantic-acoustic dual-stream quantization. By disentangling semantic and acoustic modeling into two dedicated streams, SAC enables each to be optimized for its respective role. Comprehensive evaluations show that SAC achieves strong reconstruction performance across diverse bitrates under both clean and noisy conditions, with particularly high scores on UTMOS and WER, demonstrating superior perceptual quality and intelligibility. Moreover, SAC substantially outperforms state-of-the-art codecs in semantic representation, achieving a level comparable to that of self-supervised learning (SSL) continuous embeddings. Finally, our analysis of speech disentanglement highlights the effectiveness of the dual-stream design, offering new potential for controllable speech applications.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents SAC, a novel neural speech codec that effectively disentangles semantic and acoustic information, achieving state-of-the-art performance in speech reconstruction and semantic representation. This work is a significant contribution to the field of audio processing, offering innovative methodologies and robust experimental validation that could shape future research and applications in speech technology.
Methodology
The proposed SAC architecture introduces a dual-stream quantization approach that effectively disentangles semantic and acoustic representations, allowing for specialized optimization of each stream. This is a significant methodological advancement over existing codecs that typically fuse these representations, leading to potential improvements in both semantic fidelity and reconstruction quality. The use of a pre-trained semantic tokenizer and the incorporation of speaker feature supervision are innovative strategies that enhance the model's performance in capturing linguistic content and timbre, respectively. The architecture is built on a VQ-GAN framework, which is well-suited for the task of speech reconstruction.
Results & evaluation
The experimental evaluation is comprehensive, utilizing diverse datasets and metrics to assess both reconstruction quality and semantic representation. The paper reports strong performance across various bitrates, demonstrating the robustness of SAC in both clean and noisy conditions. The results indicate that SAC outperforms state-of-the-art codecs in both speech intelligibility and semantic representation, which is validated through rigorous ablation studies. The use of benchmarks like UTMOS and WER provides a solid foundation for the claims made regarding performance improvements.
Reproducibility
The paper includes detailed descriptions of the training setup, datasets, and evaluation metrics, which are essential for reproducibility. The authors provide links to the code and pre-trained models, facilitating further research and validation of their findings. However, the paper could benefit from additional details on hyperparameter tuning and specific training configurations to enhance reproducibility.
Limitations
While SAC shows promising results, its generalizability to other audio domains, such as music or non-speech sounds, remains untested. The reliance on a speech-specific semantic tokenizer may limit the model's applicability in broader contexts. Additionally, the performance of SAC in real-world applications, such as live speech processing, has not been evaluated, which could reveal further limitations.
Broader impact
The advancements presented in SAC have significant implications for applications in speech compression, synthesis, and understanding, particularly in scenarios requiring high fidelity and intelligibility. The ability to disentangle semantic and acoustic features opens new avenues for controllable speech applications, such as voice conversion and personalized text-to-speech systems. This work could influence future research directions in audio processing and machine learning, promoting further exploration of dual-stream architectures in other domains. The paper presents SAC, a novel neural speech codec that effectively disentangles semantic and acoustic information, achieving state-of-the-art performance in speech reconstruction and semantic representation. This work is a significant contribution to the field of audio processing, offering innovative methodologies and robust experimental validation that could shape future research and applications in speech technology.
Author of this paper? Send a correction.