YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidance
Abstract
Singing Voice Synthesis (SVS) remains constrained in practical deployment due to its strong dependence on accurate phoneme-level alignment and manually annotated melody contours, requirements that are resource-intensive and hinder scalability. To overcome these limitations, we propose a melody-driven SVS framework capable of synthesizing arbitrary lyrics following any reference melody, without relying on phoneme-level alignment. Our method builds on a Diffusion Transformer (DiT) architecture, enhanced with a dedicated melody extraction module that derives melody representations directly from reference audio. To ensure robust melody encoding, we employ a teacher model to guide the optimization of the melody extractor, alongside an implicit alignment mechanism that enforces similarity distribution constraints for improved melodic stability and coherence. Additionally, we refine duration modeling using weakly annotated song data and introduce a Flow-GRPO reinforcement learning strategy with a multi-objective reward function to jointly enhance pronunciation clarity and melodic fidelity. Experiments show that our model achieves superior performance over existing approaches in both objective measures and subjective listening tests, especially in zero-shot and lyric adaptation settings, while maintaining high audio quality without manual annotation. This work offers a practical and scalable solution for advancing data-efficient singing voice synthesis. To support reproducibility, we release our inference code and model checkpoints.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a scalable, annotation-free framework for singing voice synthesis that significantly enhances the accessibility and quality of synthesized singing voices. This work represents a meaningful step forward in the field of audio synthesis, combining innovative methodologies with practical applications that could reshape how music is produced and experienced.
Methodology
The proposed methodology introduces a novel melody-driven Singing Voice Synthesis (SVS) framework that leverages a Diffusion Transformer architecture combined with a melody extraction module. This innovative approach eliminates the need for phoneme-level alignment and manual annotations, addressing significant scalability issues in existing SVS systems. The integration of a teacher model for melody extraction and the use of a similarity distribution constraint for melodic stability are particularly noteworthy. Additionally, the incorporation of reinforcement learning through Flow-GRPO for post-training optimization is a strong methodological advancement, enhancing both pronunciation clarity and melodic fidelity.
Results & evaluation
The experimental evaluation is robust, demonstrating the proposed model's superiority over existing approaches in both objective metrics (like Word Error Rate and F0 correlation) and subjective listening tests. The authors provide comprehensive comparisons across multiple experimental setups, including zero-shot synthesis and lyric editing, showcasing significant improvements in performance. The use of diverse datasets and the detailed reporting of results strengthen the credibility of the findings.
Reproducibility
The authors have taken steps to ensure reproducibility by releasing their inference code and model checkpoints, which is a positive aspect for the research community. However, the paper could benefit from more detailed descriptions of the training data preparation and specific hyperparameter settings used during experiments to further enhance reproducibility.
Limitations
While the paper presents a significant advancement, it does not address potential limitations such as the model's performance across different languages or styles beyond the Mandarin dataset used for training. Additionally, the reliance on weakly annotated data may introduce variability in performance that is not fully explored.
Broader impact
The framework has the potential to democratize music creation by enabling users without professional musical training to synthesize high-quality singing voices. This could have wide-ranging applications in the entertainment industry, including music production, virtual performances, and interactive media. The approach also opens avenues for further research in multilingual and cross-cultural singing synthesis. The main contribution of this paper is the introduction of a scalable, annotation-free framework for singing voice synthesis that significantly enhances the accessibility and quality of synthesized singing voices. This work represents a meaningful step forward in the field of audio synthesis, combining innovative methodologies with practical applications that could reshape how music is produced and experienced.
Author of this paper? Send a correction.