Adapting Diffusion-Based Music Synthesis to Speech and Singing Voice Conversion
Abstract
Recent diffusion-based generative models have achieved strong results in domain-specific audio generation tasks such as speech, singing, and instrumental music synthesis. However, these models are typically specialized and do not generalize well to mixed or intermediate audio types. In this work, we adapt a diffusion-based model originally designed for multi-instrument music synthesis to voice conversion, covering both speech and singing within a unified framework. Specifically, we extend musical note-based conditioning to include phonetic posteriorgrams (PPGs) and pitch contours, and reinterpret timbre conditioning as speaker or singer identity via feature-wise linear modulation. Experiments show that the adapted model matches or surpasses a dedicated voice conversion system in terms of naturalness and performer similarity, while maintaining accurate pitch control across speech and singing. At the same time, we observe limitations in phonetic fidelity and a degradation in vocal quality when incorporating instrumental training data. Furthermore, we demonstrate that off-the-shelf feature extractors provide effective conditioning signals, enabling large-scale self-supervised training without manual annotations. These results highlight the potential of cross-domain model transfer towards unified audio generation systems capable of handling speech, singing, and music. Qualitative samples can be found on our project page: https://benadar293.github.io/voice-conversion
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper effectively demonstrates that a diffusion model designed for instrumental music synthesis can be adapted to high-quality speech and singing voice conversion by extending its conditioning mechanisms to include phonetic and pitch information, achieving performance comparable to specialized systems while highlighting the inherent trade-offs in unified audio generation frameworks.
Methodology
The paper proposes a method to adapt a diffusion-based model originally designed for multi-instrument music synthesis (T5-based) to the task of speech and singing voice conversion (VC). The core technical contribution lies in the conditioning framework: it extends musical note-based conditioning (piano rolls) to include phonetic posteriorgrams (PPGs) and pitch contours ($f_0$) for vocal content, and reinterprets timbre conditioning via Feature-wise Linear Modulation (FiLM) as speaker/singer identity. The authors also implement pitch range adaptation to handle the disparity between source and target vocal ranges. The methodology is sound and leverages existing, robust components (CREPE for pitch, wav2vec 2.0 for PPGs, TRILL for embeddings), focusing on the architectural adaptation and conditioning strategy rather than proposing a new neural architecture from scratch. The comparison with a specialized VC model (FlowMAC) provides a strong baseline for evaluating the efficacy of the cross-domain transfer.
Results & evaluation
The experimental setup is comprehensive, covering both speech and singing domains. The authors employ a rigorous evaluation protocol including MUSHRA-style listening tests for naturalness and similarity, as well as objective metrics like Fréchet Audio Distance (FAD), Raw Pitch Accuracy (RPA), and phonetic fidelity metrics (Jensen-Shannon and Wasserstein distances on PPGs). The results indicate that the adapted music synthesis model (T5-Voc) matches or slightly surpasses the dedicated VC model (MAC-Voc) in naturalness and performer similarity, while maintaining accurate pitch control. However, the paper honestly reports limitations: phonetic fidelity is lower in the T5 model compared to the specialized model, and incorporating instrumental data (T5-All) degrades vocal quality. The inclusion of a "Vocals with Instrumentals" experiment demonstrates the potential of the unified framework, showing improvements in FAD when combining vocal and instrumental conditioning. The statistical analysis (Wilcoxon signed-rank test) adds credibility to the subjective findings.
Reproducibility
The paper provides significant detail regarding the model architecture, conditioning mechanisms, and training procedures. It specifies the use of off-the-shelf feature extractors (CREPE, wav2vec 2.0, TRILL) and mentions the use of publicly available implementations for the baseline models. The dataset description is somewhat vague regarding the proprietary speech data ("proprietary and public sources"), which limits exact replication of the training set. However, the authors state that the feature extractors are public, allowing for the creation of pseudo-annotated datasets from equivalent audio. The project page link provides qualitative samples, aiding in verification. The code is not explicitly linked in the text provided, but the reliance on public components enhances reproducibility potential.
Limitations
The paper identifies several key limitations. First, there is a trade-off between generative quality/naturalness and phonetic fidelity; the T5 model, while sounding more natural, is less precise in preserving specific phonetic content compared to the specialized FlowMAC model. Second, the degradation in vocal quality when training on mixed vocal-instrumental data (T5-All) highlights the difficulty of unified modeling without careful architectural or training interventions. Third, the reliance on source separation for mixed audio introduces potential errors, although the authors claim high-quality separation. Finally, the evaluation of phonetic fidelity relies on automatic metrics which may not fully correlate with human perception of intelligibility, a gap the authors acknowledge.
Broader impact
This work contributes to the growing field of unified audio generation models, demonstrating that domain-specific architectures can be effectively repurposed for new tasks through careful conditioning design. It highlights the potential for cross-domain transfer in audio synthesis, which could lead to more versatile and efficient generative systems. The findings also have implications for the development of foundation models for audio, suggesting that while unified models are promising, they currently face challenges in balancing performance across diverse domains (speech vs. singing vs. instrumental). The open release of qualitative samples and the discussion of self-supervised training with off-the-shelf features support the broader ML community's interest in scalable, annotation-efficient audio generation methods. This paper effectively demonstrates that a diffusion model designed for instrumental music synthesis can be adapted to high-quality speech and singing voice conversion by extending its conditioning mechanisms to include phonetic and pitch information, achieving performance comparable to specialized systems while highlighting the inherent trade-offs in unified audio generation frameworks.
Author of this paper? Send a correction.