Apollo: Unified Multi-Task Audio-Video Joint Generation
Abstract
Audio-video joint generation has progressed rapidly, yet substantial challenges still remain. Non-commercial approaches still suffer audio-visual asynchrony, poor lip-speech alignment, and unimodal degradation, which can be stemmed from weak audio-visual correspondence modeling, limited generalization, and scarce high-quality dense-caption data. To address these issues, we introduce Apollo and delve into three axes--model architecture, training strategy, and data curation. Architecturally, we adopt a single-tower design with unified DiT blocks and an Omni-Full Attention mechanism, achieving tight audio-visual alignment and strong scalability. Training-wise, we adopt a progressive multitask regime--random modality masking to joint optimization across tasks, and a multistage curriculum, yielding robust representations, strengthening A-V aligned world knowledge, and preventing unimodal collapse. For datasets, we present the first large-scale audio-video dataset with dense captions, and introduce a novel automated data-construction pipeline which annotates and filters millions of diverse, high-quality, strictly aligned audio-video-caption triplets. Building on this, Apollo scales to large datasets, delivering high-fidelity, semantically and temporally aligned, instruction-following generation in both joint and unimodal settings while generalizing robustly to out-of-distribution scenarios. Across tasks, it substantially outperforms prior methods by a large margin and achieves performance comparable to Veo 3, offering a unified, scalable path toward next-generation audio-video synthesis.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Apollo introduces a unified framework for audio-video joint generation that significantly improves alignment and generalization. The innovative methodology, combined with a large-scale dataset, positions this work as a meaningful contribution to the field of machine learning, particularly in multimedia synthesis.
Methodology
The proposed methodology in Apollo is robust, leveraging a single-tower architecture with unified DiT blocks and an Omni-Full Attention mechanism, which is innovative in achieving audio-visual alignment. The progressive multitask training regime, including random modality masking and a multistage curriculum, is a notable approach that enhances the model's ability to generalize and maintain alignment across tasks. However, the paper could benefit from a more detailed explanation of the architectural choices and their specific advantages over existing methods.
Results & evaluation
The experiments conducted demonstrate substantial improvements over prior methods, with performance metrics indicating a significant margin of advancement. The introduction of a large-scale audio-video dataset with dense captions is a commendable contribution, as it addresses a critical gap in the field. However, the paper lacks comprehensive details on the experimental setup, including specific metrics used for evaluation and comparisons against a broader range of baseline models.
Reproducibility
The paper does not provide sufficient details regarding the implementation of the model or the datasets used, which raises concerns about reproducibility. Clearer documentation of the training process, hyperparameters, and access to the dataset would enhance the ability of other researchers to replicate the results.
Limitations
While Apollo presents a unified framework, the paper does not adequately address potential limitations, such as the scalability of the model to even larger datasets or the computational resources required for training. Additionally, the impact of the automated data-construction pipeline on the quality of the generated outputs is not thoroughly examined.
Broader impact
The implications of Apollo are significant, as it paves the way for advancements in audio-video synthesis, which can have applications in various fields, including entertainment, education, and virtual reality. The ability to generate high-fidelity, temporally aligned audio-video content could revolutionize content creation and enhance user experiences in multimedia applications. Apollo introduces a unified framework for audio-video joint generation that significantly improves alignment and generalization. The innovative methodology, combined with a large-scale dataset, positions this work as a meaningful contribution to the field of machine learning, particularly in multimedia synthesis.
Author of this paper? Send a correction.