CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance
Abstract
Singing Voice Synthesis (SVS) aims to generate expressive vocal performances from structured musical inputs such as lyrics and pitch sequences. While recent progress in discrete codec-based speech synthesis has enabled zero-shot generation via in-context learning, directly extending these techniques to SVS remains non-trivial due to the requirement for precise melody control. In particular, prompt-based generation often introduces prosody leakage, where pitch information is inadvertently entangled within the timbre prompt, compromising controllability. We present CoMelSinger, a zero-shot SVS framework that enables structured and disentangled melody control within a discrete codec modeling paradigm. Built on the non-autoregressive MaskGCT architecture, CoMelSinger replaces conventional text inputs with lyric and pitch tokens, preserving in-context generalization while enhancing melody conditioning. To suppress prosody leakage, we propose a coarse-to-fine contrastive learning strategy that explicitly regularizes pitch redundancy between the acoustic prompt and melody input. Furthermore, we incorporate a lightweight encoder-only Singing Voice Transcription (SVT) module to align acoustic tokens with pitch and duration, offering fine-grained frame-level supervision. Experimental results demonstrate that CoMelSinger achieves notable improvements in pitch accuracy, timbre consistency, and zero-shot transferability over competitive baselines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
CoMelSinger presents a novel framework for zero-shot singing voice synthesis that effectively addresses melody control and prosody leakage. The combination of innovative methodology and promising experimental results positions this work as a significant contribution to the field of machine learning in audio synthesis.
Methodology
The methodology presented in CoMelSinger is innovative, leveraging a non-autoregressive MaskGCT architecture to replace traditional text inputs with discrete lyric and pitch tokens. This approach effectively addresses the challenge of prosody leakage by introducing a coarse-to-fine contrastive learning strategy, which regularizes pitch redundancy. The incorporation of a lightweight encoder-only Singing Voice Transcription (SVT) module for frame-level supervision is a significant enhancement, allowing for better alignment of acoustic tokens with pitch and duration. Overall, the methodology is well-structured and demonstrates a clear understanding of the challenges in singing voice synthesis.
Results & evaluation
The experimental setup is robust, with comprehensive evaluations against competitive baselines. The results indicate notable improvements in pitch accuracy, timbre consistency, and zero-shot transferability, which are critical metrics in the field of singing voice synthesis. However, the paper could benefit from a more detailed analysis of the datasets used and the specific metrics employed to quantify improvements, as this would enhance the credibility of the findings.
Reproducibility
While the paper outlines the methodology and experimental results, it lacks sufficient implementation details that would facilitate reproducibility. Key aspects such as hyperparameter settings, data preprocessing steps, and code availability are not mentioned, which could hinder other researchers from replicating the study.
Limitations
One limitation is the potential overfitting to the training data, particularly in the context of zero-shot learning. The paper does not address how the model performs with unseen data outside of the training distribution. Additionally, the reliance on a discrete token-based approach may limit the expressiveness of the generated singing voices compared to continuous representations.
Broader impact
The advancements made in CoMelSinger have the potential to significantly impact the fields of music technology and artificial intelligence, particularly in applications such as music composition, voice cloning, and interactive entertainment. The ability to generate expressive singing voices with structured control could lead to new creative tools for artists and musicians, enhancing the accessibility of music production. CoMelSinger presents a novel framework for zero-shot singing voice synthesis that effectively addresses melody control and prosody leakage. The combination of innovative methodology and promising experimental results positions this work as a significant contribution to the field of machine learning in audio synthesis.
Author of this paper? Send a correction.