MOVA: Towards Scalable and Synchronized Video-Audio Generation
Abstract
Audio is indispensable for real-world video, yet generation models have largely overlooked audio components. Current approaches to producing audio-visual content often rely on cascaded pipelines, which increase cost, accumulate errors, and degrade overall quality. While systems such as Veo 3 and Sora 2 emphasize the value of simultaneous generation, joint multimodal modeling introduces unique challenges in architecture, data, and training. Moreover, the closed-source nature of existing systems limits progress in the field. In this work, we introduce MOVA (MOSS Video and Audio), an open-source model capable of generating high-quality, synchronized audio-visual content, including realistic lip-synced speech, environment-aware sound effects, and content-aligned music. MOVA employs a Mixture-of-Experts (MoE) architecture, with a total of 32B parameters, of which 18B are active during inference. It supports IT2VA (Image-Text to Video-Audio) generation task. By releasing the model weights and code, we aim to advance research and foster a vibrant community of creators. The released codebase features comprehensive support for efficient inference, LoRA fine-tuning, and prompt enhancement.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MOVA, an innovative open-source model for generating high-quality synchronized video and audio content using a Mixture-of-Experts architecture. The work addresses critical challenges in the field of audio-visual generation, providing a scalable solution that could significantly enhance the quality and efficiency of content creation.
Methodology
The paper presents a novel Mixture-of-Experts (MoE) architecture for generating synchronized video and audio content. The architecture's design, which allows for a large number of parameters while activating only a subset during inference, is innovative and addresses scalability issues. The integration of audio generation with video generation in a single model is a significant advancement over traditional cascaded approaches. The methodology is well-structured, detailing the architecture, data engineering, and training strategies, which are crucial for understanding the model's capabilities.
Results & evaluation
The evaluation section appears to be comprehensive, although specific results and metrics are not detailed in the abstract. The paper claims to produce high-quality audio-visual content, including lip-synced speech and environment-aware sound effects, which suggests a rigorous experimental setup. However, without access to quantitative results or comparisons with baseline models, it is difficult to fully assess the effectiveness of the proposed method.
Reproducibility
The authors emphasize the open-source nature of their work, providing model weights and code, which is a positive aspect for reproducibility. The mention of support for efficient inference and LoRA fine-tuning indicates a thoughtful approach to making the model accessible for further research and development. However, the lack of a demo URL limits immediate accessibility for potential users.
Limitations
The paper does not explicitly discuss limitations, but potential issues could arise from the complexity of the MoE architecture, such as increased computational requirements and the challenge of fine-tuning such a large model. Additionally, the reliance on a large dataset for training may limit the model's applicability in scenarios with less available data.
Broader impact
The ability to generate synchronized audio-visual content has significant implications for various fields, including entertainment, education, and virtual reality. By releasing the model and fostering a community of creators, the authors aim to stimulate further research and innovation in this area. The open-source approach could lead to advancements in multimodal generation techniques, potentially impacting how content is created and consumed in the future. The main contribution of this paper is the introduction of MOVA, an innovative open-source model for generating high-quality synchronized video and audio content using a Mixture-of-Experts architecture. The work addresses critical challenges in the field of audio-visual generation, providing a scalable solution that could significantly enhance the quality and efficiency of content creation.
Author of this paper? Send a correction.