Geometric Iterative Retrieval for Neural Audio Codec Resynthesis
Abstract
Neural audio codecs based on Residual Vector Quantization (RVQ) have become the dominant discrete representation for token-based general audio generation, yet resynthesizing high-quality audio from coarse codec tokens remains an open problem and bounds the fidelity of every system that generates them. Prior work has framed resynthesis as a choice between discrete token prediction and continuous regression. We argue that this dichotomy is incomplete and introduce geometric iterative retrieval, a paradigm that uses the RVQ layer hierarchy itself as a natural iterative decomposition in continuous codebook space. Rather than classifying over discrete vocabularies or regressing to a single target vector, our method performs contrastive retrieval in the codebook's geometric space. We evaluate our method on codec restoration tasks across speech and music, and show improvements over both single-pass token prediction and one-step regression baselines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
Methodology
The paper proposes "Geometric Iterative Retrieval" (GIR), a novel framework for neural audio codec resynthesis. It correctly identifies a gap in the design space between discrete token prediction and continuous regression, positioning GIR in the "continuous, iterative" cell, distinct from diffusion methods by aligning its iterative steps with the RVQ layer hierarchy rather than a noise schedule. The methodology is technically sound: it uses a DeBERTa-v3 backbone to predict continuous latent vectors for each RVQ layer, supervised by a CLIP-style contrastive loss. A key innovation is the replacement of the standard additive RVQ decoding with a self-attention aggregator, allowing for non-additive combination of layer embeddings. This approach effectively leverages the geometric structure of the codebook space, addressing the "geometry-blind" nature of cross-entropy loss in token prediction and the "mean-seeking" behavior of MSE regression. The theoretical framing of the design space is clear and compelling.
Results & evaluation
The experimental setup is rigorous, evaluating on three diverse datasets (MTG-Jamendo, Common Voice, FMA) using the DAC codec. The baselines are comprehensive, including naive decoding, one-step regression (MSE and Cosine), discrete token prediction (CE), and ablations of the proposed method. The results show that GIR achieves the best Log-Spectral Distance (LSD) and is preferred in double-blind human listening tests over all baselines, including the strong one-step regression baseline. The layer-progression analysis reveals an interesting tension: objective metrics peak at K=3 layers, while human listeners prefer the full K=9 decode, suggesting current objective metrics may not fully capture perceptual quality for residual layers. This finding is valuable for the community. The ablation studies effectively isolate the contributions of the per-layer target, contrastive loss, and attention aggregator.
Reproducibility
The paper provides sufficient implementation details for reproduction, including the backbone architecture (DeBERTa-v3), training hyperparameters (AdamW, learning rate schedule, batch size), and dataset mixtures. The code repository is linked. The use of standard codecs (DAC) and metrics (LSD, SI-SDR, FAD) further aids reproducibility.
Limitations
The method is currently evaluated only on the DAC codec; generalization to other RVQ-based codecs (like EnCodec or SoundStream) is not demonstrated, though the authors claim it is codec-agnostic. The layer-progression analysis shows that adding more layers beyond K=3 degrades objective metrics, which contradicts human preference, indicating a potential misalignment in evaluation metrics or a limitation in how the model handles deep residual layers. The computational cost of the iterative inference (D-1 steps) is higher than one-step regression, though comparable to discrete token prediction.
Broader impact
This work significantly advances the field of neural audio generation by providing a high-fidelity resynthesis method that bridges the gap between discrete and continuous representations. High-quality resynthesis is critical for the fidelity of end-to-end audio generation systems. The insight that codec hierarchy can guide iterative refinement offers a new paradigm for designing generative models in other domains with hierarchical latent structures. The finding regarding the disconnect between spectral metrics and human preference has broader implications for audio evaluation benchmarking. The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
Author of this paper? Send a correction.