ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion
Abstract
Neural speech codecs efficiently compress speech and have become a foundation for speech generation, but they are typically learned as holistic representations that intertwine linguistic content, speaker identity, and prosody. While this design is effective for zero-shot voice cloning, it hinders downstream tasks that require prosody preservation or transfer, such as voice conversion. To address this, we introduce ProsoCodec, a prosody-oriented speech codec that models prosody as a conditional residual rather than as a disentangled stream. Specifically, by conditioning both the encoder and decoder on text and speaker embeddings as prefix tokens, the discrete bottleneck is encouraged to capture prosodic variation not explained by content and speaker. To further preserve prosody, we use the low-frequency mel band and train the model on paired same-speaker utterances. Experiments on voice conversion show improved prosody preservation and reduced source-timbre leakage.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ProsoCodec presents a novel and effective approach to voice conversion by treating prosody as a conditional residual in a discrete speech codec, achieving state-of-the-art performance in prosody preservation and content fidelity through clever architectural conditioning and training strategies.
Methodology
The paper proposes ProsoCodec, a neural speech codec specifically designed for voice conversion (VC) by modeling prosody as a conditional residual. The core innovation lies in the architectural conditioning: by providing explicit text and speaker embeddings as prefix tokens to both the encoder and decoder, the discrete bottleneck is forced to encode only the "residual" information, which the authors argue corresponds to prosody. This is a clever application of residual learning in the latent space. The use of Binary Spherical Quantization (BSQ) provides a compact discrete representation. The decoder utilizes a Diffusion Transformer (DiT) with flow-matching, which is a modern and effective approach for high-fidelity waveform/spectrogram reconstruction. The "dual-utterance" training strategy is a pragmatic and effective heuristic to prevent prompt-style leakage, where the model might otherwise copy the reference speaker's prosody instead of the source's. The methodology is sound, well-motivated, and builds logically on recent trends in discrete speech representation and diffusion-based generation.
Results & evaluation
The experimental setup is comprehensive, comparing ProsoCodec against several strong baselines including DDDM-VC, UniAudio, HierSpeech++, FACodec, Seed-VC, and Vevo. The evaluation covers both objective metrics (WER, Speaker Similarity, RMSE of F0, UTMOS) and subjective metrics (MOS for similarity, prosody, and naturalness). The results show that ProsoCodec achieves state-of-the-art performance in terms of content preservation (lowest WER) and prosody preservation (lowest F0 RMSE), while maintaining competitive speaker similarity and naturalness. The ablation studies are particularly valuable, demonstrating the individual contributions of the low-frequency mel input, the dual-utterance training, and the explicit conditioning. The analysis of the trade-off between bitrate and timbre leakage adds depth to the understanding of discrete codecs for VC.
Reproducibility
The paper provides detailed implementation information, including dataset (LibriTTS, VCTK), model architecture (Transformer encoder, DiT decoder), hyperparameters (learning rate, batch size, optimizer), and pre-trained models used for ASR and speaker embedding (Qwen3-ASR, CAM++). The use of standard datasets and widely available pre-trained models enhances reproducibility. However, the specific initialization from "TaDiCodec" and the exact details of the binary spherical quantization implementation could benefit from more precise citation or code availability, though the description is generally sufficient for a competent researcher to replicate.
Limitations
The paper acknowledges that strong reconstruction does not always imply strong conversion, a known issue in discrete speech models. The reliance on external ASR and speaker verification models introduces potential error propagation; if the ASR transcript is incorrect, the text conditioning will be wrong, potentially affecting prosody encoding. The dual-utterance training strategy, while effective, requires paired data from the same speaker, which might limit its applicability in scenarios where such pairs are scarce, although LibriTTS provides ample such data. The focus on prosody preservation might come at the cost of some fine-grained spectral details, as evidenced by the use of low-frequency mel bands, though the diffusion decoder helps mitigate this.
Broader impact
ProsoCodec contributes to the field of speech processing by offering a robust solution for voice conversion, a task with significant applications in entertainment, accessibility, and privacy. By decoupling prosody from timbre and content more effectively than previous holistic codecs, it enables more natural and controllable speech synthesis. The approach of using residual learning with explicit conditioning is a generalizable technique that could be applied to other speech generation tasks requiring fine-grained control over specific acoustic attributes. ProsoCodec presents a novel and effective approach to voice conversion by treating prosody as a conditional residual in a discrete speech codec, achieving state-of-the-art performance in prosody preservation and content fidelity through clever architectural conditioning and training strategies.
Author of this paper? Send a correction.