Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
Abstract
Recent progress in multimodal models has spurred rapid advances in audio understanding, generation, and editing. However, these capabilities are typically addressed by specialized models, leaving the development of a truly unified framework that can seamlessly integrate all three tasks underexplored. While some pioneering works have explored unifying audio understanding and generation, they often remain confined to specific domains. To address this, we introduce Audio-Omni, the first end-to-end framework to unify generation and editing across general sound, music, and speech domains, with integrated multi-modal understanding capabilities. Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis. To overcome the critical data scarcity in audio editing, we construct AudioEdit, a new large-scale dataset comprising over one million meticulously curated editing pairs. Extensive experiments demonstrate that Audio-Omni achieves state-of-the-art performance across a suite of benchmarks, outperforming prior unified approaches while achieving performance on par with or superior to specialized expert models. Beyond its core capabilities, Audio-Omni exhibits remarkable inherited capabilities, including knowledge-augmented reasoning generation, in-context generation, and zero-shot cross-lingual control for audio generation, highlighting a promising direction toward universal generative audio intelligence. The code, model, and dataset will be publicly released on https://zeyuet.github.io/Audio-Omni.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Audio-Omni, a unified framework for audio understanding, generation, and editing, which leverages a novel architecture and a large-scale dataset to achieve state-of-the-art performance across multiple audio tasks. This work significantly advances the field of multimodal audio processing by providing a comprehensive solution that integrates various audio capabilities into a single model, setting a new standard for future research in generative audio intelligence.
Methodology
The paper introduces Audio-Omni, a novel framework that integrates audio understanding, generation, and editing across diverse audio domains. Its architecture employs a frozen Multimodal Large Language Model (MLLM) for high-level reasoning and a trainable Diffusion Transformer (DiT) for synthesis, which is a significant advancement in unifying these tasks. The hybrid conditioning mechanism effectively separates high-level semantic inputs from low-level signal features, allowing for precise audio manipulation. The dataset construction method is also innovative, combining real-world data mining with synthetic data generation to create a large-scale dataset for instruction-guided audio editing.
Results & evaluation
The experiments are extensive and demonstrate that Audio-Omni outperforms prior unified models and matches or exceeds the performance of specialized models across various benchmarks. The use of both objective metrics (like FAD and LSD) and subjective evaluations (human ratings) provides a comprehensive assessment of the model's capabilities. The results validate the effectiveness of the proposed architecture and its ability to generalize across multiple audio tasks.
Reproducibility
The paper provides detailed implementation details, including architecture specifications, training protocols, and evaluation metrics, which enhances reproducibility. The authors commit to releasing their code, model, and dataset publicly, which is a positive step toward enabling other researchers to replicate their findings.
Limitations
While the framework shows promise, it may still face challenges in handling highly complex audio editing tasks that require nuanced understanding beyond the current capabilities of the MLLM. Additionally, the reliance on a large-scale dataset may limit accessibility for researchers without similar resources.
Broader impact
The potential applications of this work are significant, ranging from creative audio generation to practical applications in media production and accessibility technologies. However, ethical considerations regarding the misuse of generative audio technologies, such as deepfakes, must be addressed. The main contribution of this paper is the introduction of Audio-Omni, a unified framework for audio understanding, generation, and editing, which leverages a novel architecture and a large-scale dataset to achieve state-of-the-art performance across multiple audio tasks. This work significantly advances the field of multimodal audio processing by providing a comprehensive solution that integrates various audio capabilities into a single model, setting a new standard for future research in generative audio intelligence.
Author of this paper? Send a correction.