UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
Abstract
We present UNISON, a latent diffusion framework that unifies speech generation, sound generation, and audio editing within a single model. A single model handles text-to-audio, text-to-speech, zero-shot speaker cloning, mixed speech-and-sound generation, scene-level audio editing, speech-in-scene editing, and timed temporal composition, all of which share a single set of weights. Our architecture features two core designs: (1) Layer-wise deep LLM fusion, which injects hidden states from uniformly sampled layers of a frozen MLLM into corresponding MM-DiT blocks via learned projections, providing depth-matched semantic conditioning that improves instruction following over single-layer baselines; and (2) a unified multi-task architecture where task identity is encoded solely by a channel-wise mask and source audio is provided through VAE-encoded channel concatenation. Training is stabilized by an online GPU-side multi-task data synthesis pipeline with task-homogeneous batching and a two-stage curriculum. With 621M--732M trainable parameters, UNISON achieves results competitive with or exceeding task-specialist models across evaluated domains, while being roughly $4\times$ smaller than comparable unified systems.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
UNISON presents a unified framework for sound generation and editing through deep LLM fusion, significantly advancing the state of multimodal audio processing. The methodology effectively combines diverse tasks into a single model while demonstrating competitive performance against specialized systems, showcasing the potential for more efficient and scalable audio generation solutions.
Methodology
The proposed methodology in UNISON is innovative, featuring a unified architecture that integrates multiple audio generation and editing tasks into a single framework. The use of layer-wise deep LLM fusion for semantic conditioning is a significant advancement over existing models that typically rely on single-layer conditioning. The architecture's ability to handle diverse tasks with a shared latent space and a single set of weights demonstrates a thoughtful approach to reducing complexity and enhancing cross-task knowledge transfer. The online multi-task data synthesis pipeline and curriculum training further contribute to the robustness of the training process, ensuring stability and effectiveness in learning.
Results & evaluation
The experimental evaluation is comprehensive, covering a wide range of benchmarks across text-to-audio, text-to-speech, zero-shot cloning, and audio editing tasks. The results show that UNISON performs competitively against task-specialist models, achieving superior performance in several metrics such as FAD, CLAP, and WER. The ablation studies provide valuable insights into the importance of the proposed architectural choices, confirming the effectiveness of the layer-wise deep LLM fusion and the necessity of a multi-task training approach. The use of both objective and subjective metrics enhances the credibility of the findings.
Reproducibility
The paper provides detailed implementation details, including model configurations, training data composition, and hyperparameters, which are essential for reproducibility. However, the absence of a publicly available code repository limits the ability for others to replicate the results fully. The authors could enhance reproducibility by releasing their code and trained models.
Limitations
The paper acknowledges limitations related to the VAE reconstruction quality, particularly for speech, which may affect the overall output quality. Additionally, the synthetic training data for editing tasks may not fully capture the complexities of real-world audio scenes, potentially impacting the model's performance in practical applications. The current model's language support is limited to English and Chinese, which may restrict its applicability in multilingual contexts.
Broader impact
UNISON has the potential to significantly impact the fields of audio generation and editing by providing a unified framework that simplifies the deployment of audio systems. Its ability to handle multiple tasks with a single model could lead to advancements in applications such as virtual assistants, content creation, and audio post-production. The integration of LLMs into audio processing also opens avenues for more intelligent and context-aware audio systems. UNISON presents a unified framework for sound generation and editing through deep LLM fusion, significantly advancing the state of multimodal audio processing. The methodology effectively combines diverse tasks into a single model while demonstrating competitive performance against specialized systems, showcasing the potential for more efficient and scalable audio generation solutions.
Author of this paper? Send a correction.