UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions
Abstract
Generative audio modeling has largely been fragmented into specialized tasks, text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA), each operating under heterogeneous control paradigms. Unifying these modalities remains a fundamental challenge due to the intrinsic dissonance between structured semantic representations (speech/music) and unstructured acoustic textures (sound effects). In this paper, we introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects through a standardized, reference-free natural language instruction interface. To reconcile structural disparities, we propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space, enabling precise duration control within a phoneme-driven Multimodal Diffusion Transformer (MM-DiT). Coupled with a multi-stage curriculum learning strategy, this approach effectively mitigates cross-modal optimization conflicts. Extensive experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS (WER 1.47%) and TTM (SongEval Coherence 3.18), while maintaining competitive fidelity in TTA. Crucially, we observe positive transfer, where joint training on diverse audio data significantly enhances structural coherence and prosodic expressiveness compared to single-task baselines. Audio samples are available at https://qiangchunyu.github.io/UniSonate/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
UniSonate presents a unified framework for audio generation that synthesizes speech, music, and sound effects through a novel natural language interface. The technical contributions, including dynamic token injection and a multi-stage curriculum learning strategy, significantly advance the field of generative audio modeling, offering a comprehensive solution to the challenges of multimodal audio synthesis.
Methodology
The methodology proposed in UniSonate is innovative, introducing a unified flow-matching framework that integrates speech, music, and sound effect generation through a natural language interface. The dynamic token injection mechanism is particularly noteworthy as it allows unstructured sound effects to be processed in a structured manner, enabling precise control over audio generation. This is complemented by a multi-stage curriculum learning strategy that effectively mitigates optimization conflicts, showcasing a thoughtful approach to training across diverse audio modalities.
Results & evaluation
The experimental evaluation is robust, with extensive comparisons against state-of-the-art models in TTS, TTM, and TTA. The paper presents clear metrics for performance evaluation, including WER, SongEval scores, and subjective evaluations like MOS. The results indicate that UniSonate achieves state-of-the-art performance in TTS and TTM while maintaining competitive fidelity in TTA, demonstrating the effectiveness of the proposed methods.
Reproducibility
The paper provides a comprehensive description of the model architecture, training procedures, and datasets used, which supports reproducibility. However, the lack of a public code repository may hinder independent verification of results. The authors do mention the use of specific hardware configurations and hyperparameters, which aids in understanding the implementation details.
Limitations
The paper acknowledges limitations, particularly in the sound effect generation where performance lags behind specialized models. Additionally, challenges in generating long-form audio content and the inherent ambiguity in natural language instructions are highlighted. These limitations suggest areas for future research and improvement.
Broader impact
The potential applications of UniSonate are significant, as it paves the way for general-purpose audio generation systems that can synthesize complex auditory scenes. However, ethical considerations regarding the misuse of generated audio, biases in training data, and copyright issues in music generation are critical and warrant careful attention. UniSonate presents a unified framework for audio generation that synthesizes speech, music, and sound effects through a novel natural language interface. The technical contributions, including dynamic token injection and a multi-stage curriculum learning strategy, significantly advance the field of generative audio modeling, offering a comprehensive solution to the challenges of multimodal audio synthesis.
Author of this paper? Send a correction.