OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation
Abstract
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
Methodology
The paper proposes OmniVAE, a jointly trained audio-video Variational Autoencoder (VAE) designed to address the lack of cross-modal alignment in standard modality-specific tokenizers. The core methodological contribution lies in two training-only objectives added to standard reconstruction losses: (1) Segment-level Audio-Video Contrastive Learning (AVCLIP), which aligns audio and video latents at a fine temporal granularity using a hierarchical negative sampling strategy (intra-clip, sibling-clip, cross-video), and (2) Modality-Specific Semantic Distillation, which injects semantic structure from pretrained Qwen3-Omni encoders into each latent space independently. The approach is technically sound, leveraging established contrastive learning frameworks (InfoNCE) and distillation techniques (iREPA-style) but applying them specifically to the joint audio-video latent space. The three-stage training schedule (pretrain reconstruction -> joint contrastive/distillation -> adversarial refinement) is a practical engineering solution to handle the conflicting optimization dynamics of reconstruction and alignment.
Results & evaluation
The experimental evaluation is comprehensive and well-structured. The authors conduct extensive ablations to isolate the contributions of reconstruction, distillation, and contrastive learning. They employ a novel "sync probing" task (predicting temporal offset) as a proxy for alignment, which provides strong quantitative evidence that the latent spaces are indeed better aligned than baselines. Downstream text-to-audio-video generation experiments on Verse-Bench show consistent improvements in cross-modal alignment metrics (DeSync, LSE-C) and generation quality (AudioBox, IS, FAD) when using OmniVAE. The use of a large-scale, curated dataset (23M clips) with LLM-generated captions adds significant weight to the empirical claims. The reconstruction quality remains competitive with state-of-the-art independent VAEs, demonstrating that alignment does not come at the cost of fidelity.
Reproducibility
The paper provides detailed architectural descriptions, loss formulations, and training schedules. The parameter counts are explicitly listed, distinguishing between inference and training-only modules. The dataset construction pipeline is described, though the specific filtering thresholds and LLM prompts might vary. The reliance on proprietary or large pretrained models (Qwen3-Omni, Wan2.2, HunyuanVideo-Foley) for initialization and distillation is noted, which aids reproducibility for those with access to these resources. The code is not explicitly linked in the text, but the methodology is described with sufficient detail for replication.
Limitations
The primary limitation is that the alignment is learned during training only; the model does not enforce a shared latent distribution, meaning the latents are still modality-specific but aligned in a shared embedding space via the contrastive head. This might limit direct cross-modal generation without additional mapping layers. The sync probing metric, while useful, is a proxy and may not perfectly correlate with human perception of synchronization in complex scenes. The dataset, while large, is web-scraped and may contain noise despite filtering. The reliance on Qwen3-Omni for distillation assumes the availability of this specific large model, which might not be accessible to all researchers.
Broader impact
OmniVAE contributes to the field of unified multimodal generation by providing a stronger foundational representation for audio-video tasks. By decoupling the alignment learning from the generative model, it allows downstream generators to focus on content creation rather than learning synchronization from scratch. This could accelerate the development of high-quality, synchronized audio-video generative models for entertainment, education, and simulation. The emphasis on fine-grained temporal alignment addresses a key challenge in current generative AI, potentially leading to more realistic and usable synthetic media. OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
Author of this paper? Send a correction.