DiTSinger: Scaling Singing Voice Synthesis with Diffusion Transformer and Implicit Alignment
Abstract
Recent progress in diffusion-based Singing Voice Synthesis (SVS) demonstrates strong expressiveness but remains limited by data scarcity and model scalability. We introduce a two-stage pipeline: a compact seed set of human-sung recordings is constructed by pairing fixed melodies with diverse LLM-generated lyrics, and melody-specific models are trained to synthesize over 500 hours of high-quality Chinese singing data. Building on this corpus, we propose DiTSinger, a Diffusion Transformer with RoPE and qk-norm, systematically scaled in depth, width, and resolution for enhanced fidelity. Furthermore, we design an implicit alignment mechanism that obviates phoneme-level duration labels by constraining phoneme-to-acoustic attention within character-level spans, thereby improving robustness under noisy or uncertain alignments. Extensive experiments validate that our approach enables scalable, alignment-free, and high-fidelity SVS.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents DiTSinger, a novel approach to Singing Voice Synthesis that effectively scales model and data while improving alignment robustness, marking a significant contribution to the field of audio synthesis. The innovative methodology and strong experimental validation position this work as a valuable resource for future research and applications in music technology.
Methodology
The paper introduces a two-stage data construction pipeline that effectively addresses the challenges of data scarcity and model scalability in Singing Voice Synthesis (SVS). By leveraging a compact seed set of human-sung recordings paired with LLM-generated lyrics, the authors create a large-scale dataset that enhances phonetic coverage and melodic alignment. The proposed Diffusion Transformer (DiTSinger) incorporates novel architectural elements like rotary positional encoding (RoPE) and qk-norm, which are systematically scaled for improved fidelity. Additionally, the implicit alignment mechanism is a significant innovation, allowing the model to operate without phoneme-level duration labels, thus enhancing robustness against timing variability. This methodology is well-structured and demonstrates a clear understanding of the challenges in the field.
Results & evaluation
The experiments are extensive and well-designed, utilizing a dataset of over 500 hours of singing data from professional vocalists. The evaluation metrics, including MCD, FFE, and F0RMSE, are appropriate for assessing the quality of the synthesized singing. The comparisons with state-of-the-art methods, such as DiffSinger and StyleSinger, show that DiTSinger achieves superior performance, particularly in subjective measures like MOS. However, the paper could benefit from more detailed ablation studies to further validate the contributions of individual components.
Reproducibility
The paper provides sufficient implementation details, including training configurations, dataset sizes, and evaluation protocols, which are crucial for reproducibility. However, the lack of a publicly accessible code repository or demo URL limits the practical reproducibility of the results. Future work should consider releasing the model and code to facilitate community engagement and validation.
Limitations
The primary limitation of this work is its focus on Chinese singing data, which may restrict the generalizability of the findings to other languages or singing styles. Additionally, the model does not account for various singing techniques, which could impact the quality of synthesized voices in diverse musical contexts. The authors acknowledge these limitations and suggest future work to expand the dataset and incorporate additional conditions.
Broader impact
The advancements in SVS presented in this paper have significant implications for the music industry, particularly in areas such as music production, entertainment, and education. The ability to generate high-fidelity singing voices from text opens new avenues for creative expression and accessibility in music creation. Furthermore, the methodologies developed could be adapted for other audio synthesis tasks, broadening the impact of this research beyond singing voice synthesis. The paper presents DiTSinger, a novel approach to Singing Voice Synthesis that effectively scales model and data while improving alignment robustness, marking a significant contribution to the field of audio synthesis. The innovative methodology and strong experimental validation position this work as a valuable resource for future research and applications in music technology.
Author of this paper? Send a correction.