OmniCodec: Low Frame Rate Universal Audio Codec with Semantic-Acoustic Disentanglement
Abstract
Large Language Models (LLMs) have advanced audio generation through discrete representation learning. However, most existing neural codecs focus on speech and emphasize reconstruction fidelity, overlooking unified low frame rate modeling across diverse audio domains, including speech, music, and general sound. Moreover, high reconstruction quality does not necessarily yield semantically informative representations, limiting effectiveness in downstream generation tasks. We propose OmniCodec, a universal neural audio codec tailored for low frame rate. It adopts a hierarchical multi-codebook design with semantic-acoustic decoupling by leveraging the audio encoder of the pre-trained understanding model, along with a self-guidance strategy to improve codebook utilization and reconstruction. Compared with the Mimi codec, experiments show that OmniCodec achieves outstanding performance at the same bitrate, delivering superior reconstruction quality while also providing more semantically informative representations that benefit downstream generation tasks. Our model and code will be open-sourced. Our demo page is available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of OmniCodec, a universal neural audio codec that effectively combines low frame rate modeling with semantic-acoustic decoupling, achieving superior reconstruction quality and semantic representation across diverse audio domains. This work significantly advances the state of audio codecs, particularly in their application to large language models and generative tasks, and sets a foundation for future research in audio representation learning.
Methodology
The methodology presented in this paper is innovative, particularly with its hierarchical multi-codebook design and the semantic-acoustic decoupling approach. The use of a pre-trained understanding model's audio encoder to enhance semantic representation is a novel contribution that addresses the limitations of existing codecs. The self-guidance strategy to improve codebook utilization is also a noteworthy addition, demonstrating a thoughtful approach to enhancing reconstruction quality while maintaining low frame rates.
Results & evaluation
The experiments are robust, utilizing a comprehensive dataset of approximately 160,000 hours of audio across various domains (speech, music, and general sound). The evaluation metrics are well-chosen, including both objective measures (PESQ, STOI, Mel distance) and subjective assessments (N-MOS, S-MOS). The results indicate that OmniCodec outperforms existing models, particularly in the music and general sound domains, which validates the effectiveness of the proposed architecture.
Reproducibility
The paper provides sufficient implementation details, including model architecture, training procedures, and hyperparameters, which facilitates reproducibility. The open-sourcing of the model and code further enhances the potential for other researchers to replicate and build upon this work.
Limitations
One limitation noted is the performance disparity in the speech domain compared to other models, which may be attributed to the structure of the WavLM model used for semantic supervision. Additionally, the paper acknowledges challenges in achieving optimal semantic decoupling for speech, suggesting that future work will be needed to address these issues.
Broader impact
The proposed OmniCodec has significant implications for audio generation tasks across various domains, including speech synthesis and music generation. Its ability to provide semantically informative representations can enhance applications in multimedia content creation, real-time audio processing, and interactive systems. The open-source nature of the project encourages further exploration and innovation in the field. The main contribution of this paper is the introduction of OmniCodec, a universal neural audio codec that effectively combines low frame rate modeling with semantic-acoustic decoupling, achieving superior reconstruction quality and semantic representation across diverse audio domains. This work significantly advances the state of audio codecs, particularly in their application to large language models and generative tasks, and sets a foundation for future research in audio representation learning.
Author of this paper? Send a correction.