AugCodec: A Low-Bitrate Disentangled Neural Speech Codec via Data Augmentation
Abstract
We propose AugCodec, a low-bitrate disentangled neural speech codec that leverages data augmentation to decompose speech into three distinct components: semantic, speaker, and prosody tokens. Specifically, we employ tailored augmenta tion strategies to transform speech into distinct variants, each serving as input for extracting tokens that preserve the target attribute while suppressing others. This disentanglement strategy enables substantial reduction in token rate. Further more, we introduce an augmentation loss that aligns semantic encoder outputs between source and voice-converted speech, encouraging speaker-agnostic embeddings while mitigating the acoustic mismatch induced by voice conversion. Experiments on LibriSpeech test-clean demonstrate that AugCodec significantly outperforms state-of-the-art methods in both reconstruction quality and disentanglement, while operating at only 12.5Hz with three token streams.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AugCodec introduces a data-augmentation-driven disentanglement strategy for neural speech codecs, achieving state-of-the-art reconstruction and content preservation at low bitrates by isolating semantic, speaker, and prosody features through tailored input transformations and alignment losses.
Methodology
The paper proposes AugCodec, a novel approach to disentangled neural speech coding that leverages data augmentation as a primary mechanism for feature separation. The core innovation lies in the input preprocessing strategies: using a voice-converted speech variant for the semantic encoder to enforce speaker invariance, a different utterance from the same speaker for the speaker encoder to isolate identity, and low-frequency STFT components for the prosody encoder to capture F0 and lower harmonics while discarding timbral content. This is complemented by an "augmentation loss" that aligns semantic embeddings between the original and voice-converted speech, explicitly penalizing speaker information leakage. The architecture employs separate streams for semantic, speaker, and prosody tokens, which are quantized independently (using VQ for semantic, FSQ for speaker/prosody) and fused via a learned expansion and modulation mechanism (FiLM) before decoding. The semantic encoder uses a learned compression/expansion scheme to reduce frame rate while preserving dynamics, addressing a key limitation of average pooling.
Results & evaluation
The authors evaluate AugCodec on LibriSpeech test-clean, comparing against strong baselines including BiCodec, Mimi, Qwen-TTS-Tokenizer-12Hz, and FACodec. The results demonstrate that AugCodec achieves superior reconstruction quality (PESQ, UTMOS) and significantly lower Word Error Rates (WER) compared to baselines at comparable or lower bitrates (e.g., 387.5 bps for AugCodec-2 vs 412.5 bps for Mimi). The disentanglement capability is validated through voice conversion experiments, where AugCodec shows better content preservation (lower WER) than FACodec and BiCodec. The ablation study confirms the importance of the augmentation loss, showing performance degradation when it is removed. The results are robust across different codebook sizes and frame rates.
Reproducibility
The paper provides detailed architectural specifications, including layer dimensions, kernel sizes, and loss weights. It specifies the use of open-source components like wav2vec 2.0, ECAPA-TDNN, and Seed-VC. The training setup (optimizer, learning rate, batch size, iterations) is clearly described. However, the specific random seeds and exact data splitting procedures for the 3000 hours of LibriLight-medium data are not fully detailed, which might introduce minor variability. The use of off-the-shelf voice conversion models introduces a dependency that is well-documented but adds complexity to the training pipeline.
Limitations
The reliance on an external voice conversion model (Seed-VC) for generating semantic training data is a potential limitation, as errors or biases in the VC model could propagate to the codec. The prosody encoder's reliance on low-frequency STFT components might struggle with complex harmonic structures or high-frequency prosodic cues, although the authors argue this minimizes correlation with semantic/speaker features. The method operates at a maximum frame rate of 12.5Hz for semantic tokens, which is low but may not be sufficient for all real-time applications requiring ultra-low latency without further optimization. The paper does not report results on out-of-domain datasets or diverse languages, limiting the generalizability assessment.
Broader impact
AugCodec contributes to the development of efficient, disentangled speech representations, which are foundational for speech language models, voice conversion, and text-to-speech systems. By enabling low-bitrate transmission with high quality and disentanglement, it facilitates more scalable and privacy-preserving speech applications. The emphasis on speaker invariance in semantic tokens has implications for reducing bias and protecting speaker identity in downstream tasks. AugCodec introduces a data-augmentation-driven disentanglement strategy for neural speech codecs, achieving state-of-the-art reconstruction and content preservation at low bitrates by isolating semantic, speaker, and prosody features through tailored input transformations and alignment losses.
Author of this paper? Send a correction.