UniVoice: A Unified Model for Speech and Singing Voice Generation
Abstract
Text-to-speech (TTS) and singing voice synthesis (SVS) both aim to generate human vocal audio from symbolic inputs, but they impose different requirements on the generation process. Speech generation relies on flexible, language-driven prosody, whereas singing generation requires explicit melody control and accurate rhythmic alignment. This mismatch makes it challenging to train a single model that can generate both natural speech and controllable singing, since melody-related conditions should strongly constrain singing but should not restrict speech prosody. We present UniVoice, a unified speech and singing voice generation framework based on conditional flow matching. Instead of using a single undifferentiated conditioning representation, UniVoice factorizes the condition into content, melody, and timbre, which are encoded by modality-appropriate encoders and consumed by a shared Diffusion Transformer (DiT) backbone. For singing, the melody condition is represented by MIDI note sequences; for speech, it is replaced with a learned null melody token, allowing the model to infer prosody from linguistic and acoustic context. This design preserves explicit melody control for singing while avoiding the need to impose melody constraints on speech. We further analyze the null melody token as an approximation to melody marginalization in the conditional flow. Trained on 30k hours of speech and 35k hours of singing data, UniVoice achieves a speech PER of 5.26\%, comparable to dedicated TTS systems such as F5-TTS (5.21\%) and CosyVoice3 (5.30\%). On singing generation, UniVoice achieves a PER of 16.22\%, outperforming the unified baseline Vevo1.5 (24.72\%).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
UniVoice presents a unified framework for speech and singing voice generation that effectively combines the requirements of both modalities through innovative conditioning strategies. The paper makes a substantial contribution to the field of audio synthesis, showcasing significant advancements in methodology and experimental validation, while also addressing broader societal implications.
Methodology
The methodology presented in UniVoice is innovative, particularly in its use of conditional flow matching and a shared Diffusion Transformer backbone to unify speech and singing voice generation. The factorization of conditioning into content, melody, and timbre is a significant advancement, allowing for effective separation of the requirements for TTS and SVS. The introduction of a learned null melody token for speech generation is particularly noteworthy, as it addresses the challenge of conflicting conditioning semantics. The architecture is well-justified theoretically, and the design choices are clearly articulated, making the methodology robust and sound.
Results & evaluation
The experiments are comprehensive, utilizing a large-scale dataset of 65k hours of audio, which is a significant contribution to the field. The results demonstrate competitive performance in both speech and singing generation, with detailed evaluations using multiple metrics, including PER, SIM, S-MOS, and N-MOS. The ablation studies provide strong evidence for the importance of the proposed components, particularly factorized conditioning and the null melody token. The introduction of the UniSinging-Eval benchmark further enhances the experimental evaluation by addressing gaps in existing evaluation protocols.
Reproducibility
The authors commit to releasing inference code, model checkpoints, and the UniSinging-Eval test set, which is crucial for reproducibility. The detailed description of the training procedure, data sources, and evaluation metrics also supports reproducibility efforts. However, the lack of a specific GitHub repository or direct access to code may hinder some aspects of reproducibility.
Limitations
While the paper presents a strong framework, it acknowledges limitations such as the SIM gap compared to dedicated speech systems and the focus on English and Chinese, which may limit multilingual applicability. The model's performance in mixed vocal styles like rap and spoken singing is not explicitly modeled, which could be a potential area for improvement. Additionally, the higher latency introduced by the 32-step ODE sampling may affect real-time applications.
Broader impact
The potential applications of UniVoice are significant, particularly in accessibility, music creation, and education. It could empower users with vocal impairments to produce expressive vocal content and assist musicians in rapid prototyping. However, the risks associated with deepfakes and copyright issues must be addressed, and the authors suggest measures such as watermarking and consent-based enrollment to mitigate these risks. UniVoice presents a unified framework for speech and singing voice generation that effectively combines the requirements of both modalities through innovative conditioning strategies. The paper makes a substantial contribution to the field of audio synthesis, showcasing significant advancements in methodology and experimental validation, while also addressing broader societal implications.
Author of this paper? Send a correction.