UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstruction
Abstract
Neural Audio Codecs (NACs) can reduce transmission overhead by performing compact compression and reconstruction, which also aim to bridge the gap between continuous and discrete signals. Existing NACs can be divided into two categories: multi-codebook and single-codebook codecs. Multi-codebook codecs face challenges such as structural complexity and difficulty in adapting to downstream tasks, while single-codebook codecs, though structurally simpler, suffer from low-fidelity, ineffective modeling of unified audio, and an inability to support modeling of high-frequency audio. We propose the UniSRCodec, a single-codebook codec capable of supporting high sampling rate, low-bandwidth, high fidelity, and unified. We analyze the inefficiency of waveform-based compression and introduce the time and frequency compression method using the Mel-spectrogram, and cooperate with a Vocoder to recover the phase information of the original audio. Moreover, we propose a sub-band reconstruction technique to achieve high-quality compression across both low and high frequency bands. Subjective and objective experimental results demonstrate that UniSRCodec achieves state-of-the-art (SOTA) performance among cross-domain single-codebook codecs with only a token rate of 40, and its reconstruction quality is comparable to that of certain multi-codebook methods. Our demo page is available at https://wxzyd123.github.io/unisrcodec.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents UniSRCodec, a unified and low-bitrate single-codebook neural audio codec that employs innovative techniques for high-fidelity audio modeling. Its contributions to the field of audio processing are substantial, addressing critical challenges in codec design and demonstrating impressive performance across diverse audio domains.
Methodology
The paper introduces the UniSRCodec, a novel single-codebook neural audio codec that addresses the limitations of existing codecs by employing a time and frequency compression method using Mel-spectrograms and a sub-band reconstruction technique. The methodology is well-structured, with a clear explanation of the encoder-quantizer-decoder architecture and the rationale behind using Mel-spectrograms. The sub-band reconstruction approach is particularly innovative, allowing for improved modeling of both low and high-frequency audio signals. The paper also discusses the training procedure and the use of various loss functions effectively, contributing to the overall robustness of the proposed method.
Results & evaluation
The experimental evaluation is thorough, utilizing a diverse set of cross-domain datasets that include speech, music, and general sound. The results demonstrate that UniSRCodec achieves state-of-the-art performance among single-codebook codecs, with a low token rate of 40 and a bitrate of 0.52 kbps. The paper includes both subjective and objective evaluations, providing a comprehensive assessment of the codec's performance. The ablation studies further validate the effectiveness of the proposed components, such as the sub-band reconstruction and the discriminator.
Reproducibility
The paper provides sufficient details regarding the architecture, training procedure, and datasets used, which aids in reproducibility. However, the lack of a public code repository or detailed implementation instructions may hinder complete reproducibility by other researchers.
Limitations
One limitation noted is that while the codec performs well across various audio domains, its performance in the speech domain is somewhat lower compared to existing models like UniCodec, primarily due to differences in training data. Additionally, the reliance on a relatively small number of GPUs for training may limit accessibility for some researchers.
Broader impact
The UniSRCodec has significant implications for audio transmission and processing, particularly in low-bitrate scenarios. Its ability to maintain high fidelity while reducing bandwidth requirements could enhance applications in streaming services, telecommunications, and audio processing in resource-constrained environments. The codec's design could also facilitate advancements in audio understanding tasks, potentially benefiting various machine learning applications. The paper presents UniSRCodec, a unified and low-bitrate single-codebook neural audio codec that employs innovative techniques for high-fidelity audio modeling. Its contributions to the field of audio processing are substantial, addressing critical challenges in codec design and demonstrating impressive performance across diverse audio domains.
Author of this paper? Send a correction.