Omni2Sound: Towards Unified Video-Text-to-Audio Generation
Abstract
Training a unified model integrating video-to-audio (V2A), text-to-audio (T2A), and joint video-text-to-audio (VT2A) generation offers significant application flexibility, yet faces two unexplored foundational challenges: (1) the scarcity of high-quality audio captions with tight A-V-T alignment, leading to severe semantic conflict between multimodal conditions, and (2) cross-task and intra-task competition, manifesting as an adverse V2A-T2A performance trade-off and modality bias in the VT2A task. First, to address data scarcity, we introduce SoundAtlas, a large-scale dataset (470k pairs) that significantly outperforms existing benchmarks and even human experts in quality. Powered by a novel agentic pipeline, it integrates Vision-to-Language Compression to mitigate visual bias of MLLMs, a Junior-Senior Agent Handoff for a 5 times cost reduction, and rigorous Post-hoc Filtering to ensure fidelity. Consequently, SoundAtlas delivers semantically rich and temporally detailed captions with tight V-A-T alignment. Second, we propose Omni2Sound, a unified VT2A diffusion model supporting flexible input modalities. To resolve the inherent cross-task and intra-task competition, we design a three-stage multi-task progressive training schedule that converts cross-task competition into joint optimization and mitigates modality bias in the VT2A task, maintaining both audio-visual alignment and off-screen audio generation faithfulness. Finally, we construct VGGSound-Omni, a comprehensive benchmark for unified evaluation, including challenging off-screen tracks. With a standard DiT backbone, Omni2Sound achieves unified SOTA performance across all three tasks within a single model, demonstrating strong generalization across benchmarks with heterogeneous input conditions. The project page is at https://swapforward.github.io/Omni2Sound.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a significant advancement in unified video-text-to-audio generation through the introduction of the SoundAtlas dataset and the Omni2Sound model, addressing critical challenges in the field. The comprehensive methodology and robust experimental evaluation highlight its potential impact on future research and applications in audio generation.
Methodology
The paper presents a comprehensive methodology that addresses two foundational challenges in unified audio generation: data scarcity and task competition. The introduction of the SoundAtlas dataset, which significantly enhances the quality and alignment of audio captions, is a notable contribution. The Omni2Sound model employs a diffusion-based architecture and a three-stage progressive training schedule, effectively mitigating cross-task and intra-task competition. The methodology is well-structured, leveraging advanced multimodal techniques and a decoupled training approach that enhances audio-visual alignment and generation fidelity.
Results & evaluation
The experimental evaluation is robust, utilizing a large-scale dataset and a comprehensive benchmark (VGGSound-Omni) to assess the performance of the Omni2Sound model across multiple tasks (V2A, T2A, VT2A). The results demonstrate state-of-the-art performance, with significant improvements over existing models. The paper includes extensive ablation studies that validate the proposed methods and their effectiveness in resolving competition between tasks. However, the paper could benefit from more detailed discussions on the statistical significance of the results.
Reproducibility
The paper provides a clear description of the methodologies and datasets used, but it lacks specific implementation details that would facilitate reproducibility. While the authors mention the use of a standard DiT backbone, further information on hyperparameters, training procedures, and code availability would enhance reproducibility.
Limitations
One limitation is the reliance on the quality of the SoundAtlas dataset, which, while claimed to outperform existing datasets, may still have inherent biases or limitations that could affect the model's performance. Additionally, the paper does not address potential scalability issues when applying the model to larger datasets or real-world applications.
Broader impact
The proposed unified model has significant implications for various applications in audio generation, including film production, video games, and virtual reality, where high-quality audio generation from multimodal inputs is crucial. The advancements in audio-visual alignment and the introduction of a comprehensive benchmark could drive further research in the field, fostering innovation in multimodal AI systems. The paper presents a significant advancement in unified video-text-to-audio generation through the introduction of the SoundAtlas dataset and the Omni2Sound model, addressing critical challenges in the field. The comprehensive methodology and robust experimental evaluation highlight its potential impact on future research and applications in audio generation.
Author of this paper? Send a correction.