EuleroDec: A Complex-Valued RVQ-VAE for Efficient and Robust Audio Coding
Abstract
Audio codecs power discrete music generative modelling, music streaming, and immersive media by shrinking PCM audio to bandwidth-friendly bitrates. Recent works have gravitated towards processing in the spectral domain; however, spectrogram domains typically struggle with phase modeling, which is naturally complex-valued. Most frequency-domain neural codecs either disregard phase information or encode it as two separate real-valued channels, limiting spatial fidelity. This entails the need to introduce adversarial discriminators at the expense of convergence speed and training stability to compensate for the inadequate representation power of the audio signal. In this work we introduce an end-to-end complex-valued RVQ-VAE audio codec that preserves magnitude-phase coupling across the entire analysis-quantization-synthesis pipeline and removes adversarial discriminators and diffusion post-filters. Without GANs or diffusion, we match or surpass much longer-trained baselines in-domain and reach SOTA out-of-domain performance on phase coherence and waveform fidelity. Compared to standard baselines that train for hundreds of thousands of steps, our model, which reduces the training budget by an order of magnitude, is markedly more compute-efficient while preserving high perceptual quality.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents EuleroDec, the first end-to-end complex-valued RVQ-VAE audio codec, which achieves state-of-the-art performance while significantly reducing training time and avoiding adversarial training methods. The methodology is innovative, addressing critical challenges in audio coding, and its implications for the field are substantial, particularly in enhancing audio quality and efficiency in various applications.
Methodology
The paper introduces a novel end-to-end complex-valued RVQ-VAE audio codec, which effectively preserves magnitude-phase coupling throughout the analysis-quantization-synthesis pipeline. The methodology is robust, employing complex convolutions, attention mechanisms, and normalization techniques that respect the algebraic structure of the STFT. The use of Wirtinger calculus for optimization is a significant advancement, allowing the model to operate entirely within the complex domain without resorting to real-valued detours. This approach addresses the limitations of existing codecs that either ignore phase information or require adversarial training, thus enhancing both convergence speed and stability.
Results & evaluation
The experiments are well-structured, benchmarking the proposed codec against established state-of-the-art models like APCodec, AudioDec, and EnCodec. The evaluation metrics are comprehensive, covering SI-SDR, PESQ, STOI, and GDD, which provide a holistic view of the codec's performance across different aspects of audio fidelity. The results demonstrate that EuleroDec achieves competitive performance with significantly fewer training steps, showcasing its efficiency and robustness in various scenarios, including out-of-domain testing.
Reproducibility
The paper provides a detailed account of the training setup, including the dataset, optimization parameters, and evaluation metrics, which enhances reproducibility. However, the absence of a publicly available implementation or code repository limits the ability for others to replicate the results directly. Future work should consider releasing the model and training code to facilitate broader validation of the findings.
Limitations
While the paper presents a significant advancement, it does not address potential limitations in terms of scalability to higher bitrates or real-time applications. Additionally, the reliance on complex-valued representations may introduce challenges in deployment on hardware that is optimized for real-valued computations. The performance in highly noisy environments or with diverse audio sources remains to be fully explored.
Broader impact
The proposed codec has the potential to significantly impact audio coding applications, particularly in scenarios requiring high fidelity at low bitrates, such as streaming and immersive media. Its efficiency and robustness could lead to advancements in music generation and speech synthesis, making it a valuable contribution to the field of audio processing. This paper presents EuleroDec, the first end-to-end complex-valued RVQ-VAE audio codec, which achieves state-of-the-art performance while significantly reducing training time and avoiding adversarial training methods. The methodology is innovative, addressing critical challenges in audio coding, and its implications for the field are substantial, particularly in enhancing audio quality and efficiency in various applications.
Author of this paper? Send a correction.