HARP: Harmonic-Aware Residual Partitioning for Neural Audio Codecs
Abstract
Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
Methodology
The paper proposes HARP, a training-time strategy for Residual Vector Quantization (RVQ) in neural audio codecs. The core innovation lies in partitioning RVQ stages into frequency-ordered groups and enforcing spectral specialization through a novel loss function. Key components include: 1) Cumulative Decoding: Ensuring higher-frequency stages are reconstructed in the context of lower-frequency content to preserve harmonic coherence. 2) Subband Contribution Supervision: Supervising the isolated contribution of each group to its target band using stop-gradient operators to prevent gradient interference. 3) Soft Band Weighting: Using learnable Gaussian weights over mel bins to guide specialization without hard spectral cutoffs. The approach is architecturally agnostic, requiring only modifications to the training objective. This is a clever and relatively simple modification that addresses the "spectral entanglement" problem inherent in standard RVQ, where early and late stages capture similar frequency content.
Results & evaluation
The authors evaluate HARP against standard DAC (same architecture, no band supervision) and BSCodec (parallel band decomposition). They use a diverse dataset including MUSDB18-HQ, LibriTTS, and AudioSet. Metrics include SI-SDR, KAD, PESQ, STOI, and MUSHRA listening tests. Results show consistent improvements in SI-SDR and KAD over both baselines, with particularly notable gains at low bitrates (2.6 kbps). The ablation studies effectively isolate the contributions of cumulative decoding, soft boundaries, and stop-gradient mechanisms. The harmonic coherence analysis using synthetic tones provides strong evidence for the preservation of phase/amplitude relationships. The MUSHRA tests confirm perceptual improvements, especially in bass fullness and timbral naturalness.
Reproducibility
The paper provides detailed descriptions of the architecture, training objectives, and hyperparameters. The code is made publicly available on GitHub. The use of standard components (DAC backbone, EMA updates) and clear algorithmic descriptions enhances reproducibility. The training cost increase (2-3x) is noted, which is a practical consideration but does not hinder reproducibility.
Limitations
The primary limitation is that the performance gains, while consistent, are modest in absolute terms (e.g., +0.25 to +0.96 dB SI-SDR). The method relies on the assumption that natural audio has a specific energy distribution (low-to-high), which might not hold for all types of audio (e.g., highly inharmonic or noise-heavy content, though the paper argues it still applies). The method is training-only, so it does not offer architectural efficiency gains. The "soft" boundaries mean that some spectral leakage or imperfect separation might still occur, potentially limiting the maximum potential of the specialization compared to hard architectural splits (though the paper argues this is offset by coherence).
Broader impact
This work contributes to the field of neural audio compression by providing a principled way to impose spectral structure on RVQ without architectural complexity. This can lead to more robust and interpretable audio codecs, which are foundational for speech synthesis, music generation, and audio understanding tasks. The improved low-bitrate performance is particularly relevant for streaming applications. The open-source release facilitates further research in this area. HARP introduces a training-only strategy for neural audio codecs that partitions RVQ stages into frequency-ordered groups with cumulative decoding and soft subband supervision, effectively addressing spectral entanglement and improving both objective metrics and perceptual quality, particularly at low bitrates, without architectural changes.
Author of this paper? Send a correction.