High Fidelity Neural Audio Compression

High Fidelity Neural Audio Compression (EnCodec)

Alexandre Défossez, Jade Copet, Gabriel Synnaeve ...

Abstract

We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.

AI research summary

The main contribution of this paper is the development of a high-fidelity neural audio codec that leverages innovative training techniques and architectures to achieve superior audio quality at low bitrates. This work significantly advances the field of audio compression by integrating neural network methodologies with practical applications in real-time audio streaming.

AI methodology analysis

The paper presents a novel audio codec that employs a streaming encoder-decoder architecture with quantized latent space, which is trained end-to-end. The introduction of a multiscale spectrogram adversary to reduce artifacts and enhance audio quality is a significant methodological advancement. The novel loss balancer mechanism that decouples the choice of hyper-parameters from the scale of the loss is particularly innovative and addresses a common challenge in training neural networks. The exploration of lightweight Transformer models for further compression also adds to the methodological depth.

AI results analysis

The authors conduct extensive experiments, including MUSHRA tests for subjective evaluation and objective metrics for performance assessment. The ablation studies provide insights into the impact of various components of the model, such as the discriminator setup and the effect of the balancer. The results show that the proposed model outperforms traditional codecs like Opus and EVS across various audio domains, demonstrating its effectiveness and robustness.

AI reproducibility analysis

The paper includes a detailed description of the model architecture, training procedures, and datasets used, which enhances reproducibility. The availability of code and models on GitHub further supports this aspect, allowing other researchers to replicate the experiments.

AI limitations analysis

While the paper presents strong results, it does not extensively address the potential limitations of the model in terms of computational efficiency at higher sample rates or the scalability of the approach for larger datasets. The reliance on subjective evaluation methods may also introduce variability in results based on listener biases.

AI broader impact analysis

The research addresses the growing need for efficient audio compression methods, particularly as internet traffic continues to rise. By improving audio quality at low bitrates, the proposed codec could enhance user experiences in streaming and communication applications, making technology more accessible in low-bandwidth scenarios. The main contribution of this paper is the development of a high-fidelity neural audio codec that leverages innovative training techniques and architectures to achieve superior audio quality at low bitrates. This work significantly advances the field of audio compression by integrating neural network methodologies with practical applications in real-time audio streaming.