Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space
Abstract
Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[Flowley introduces a novel end-to-end flow-based architecture with Progressive Soft-masked Cross-Attention for precise video-to-audio generation, achieving state-of-the-art performance on VGGSound and outperforming larger closed-source models in zero-shot settings when augmented with its SoundCap captioning pipeline.] This paper presents a significant advancement in V2A generation by eliminating the need for multi-stage training and external alignment modules, offering a more efficient and effective solution for synthesizing temporally synchronized and semantically consistent audio from video.
Methodology
The paper proposes Flowley, an end-to-end flow-based architecture for Video-to-Audio (V2A) generation. The core technical contribution is the Progressive Soft-masked Cross-Attention (PSCA) mechanism, which embeds temporal alignment directly into the attention layers of a DiT-like backbone. This avoids the need for external synchronization modules or multi-stage training pipelines common in prior works (e.g., MMAudio, VinTAGe). The method combines visual features (CLIP), textual prompts (FLAN-T5), and audio latents in a multi-stream to single-stream architecture. Additionally, the authors introduce SoundCap, a pipeline using AV-LLMs to generate detailed, sound-aware captions to address the lack of rich textual annotations in V2A datasets. The approach is technically sound, leveraging recent advances in flow matching and multimodal transformers. The PSCA mechanism is a clever, low-cost modification to standard cross-attention that explicitly models the temporal relationship between video frames and audio tokens.
Results & evaluation
The evaluation is comprehensive, covering distribution matching (KAD, FAD), audio quality (IS), semantic alignment (IB-Score, LB-Score), and synchronization (Align Acc). Flowley achieves state-of-the-art results on the VGGSound benchmark, outperforming larger models like MMAudio and FoleyCrafter in most metrics. The inclusion of SoundCap further boosts performance, particularly in zero-shot settings on the Movie Gen Audio benchmark, where it surpasses the closed-source Movie Gen Audio model in quality despite being significantly smaller and trained on less data. The subjective human evaluation supports the objective metrics, showing a preference for Flowley's synchronization and quality. The ablation studies effectively validate the contributions of PSCA, the dual cross-attention, and the SoundCap pipeline.
Reproducibility
The paper provides detailed implementation details, including architecture dimensions, hyperparameters for PSCA (window sizes, fade zones), training schedules, and evaluation metrics. The use of standard datasets (VGGSound) and established baselines facilitates comparison. The code link is provided. The description of the SoundCap pipeline is sufficiently detailed for reproduction, assuming access to the specified AV-LLMs and VLMs.
Limitations
The paper acknowledges that while Flowley excels in quality and distribution matching, it slightly lags behind some baselines in the Align Acc metric, although human preference favors its synchronization. This suggests a potential discrepancy between the automated Align Acc metric and human perception of temporal alignment. The reliance on AV-LLMs for SoundCap introduces a dependency on large, potentially expensive models for data generation, although the final VLM is lightweight. The method's performance on highly complex scenes with multiple simultaneous sound sources or off-screen sounds might be challenging, as indicated by the noise handling strategies in SoundCap.
Broader impact
This work contributes to the field of generative AI for multimedia, specifically in creating realistic and synchronized audio-visual experiences. It has applications in film production (Foley), gaming, virtual reality, and accessibility. The SoundCap pipeline offers a generalizable solution for improving text-audio grounding in other multimodal tasks. The open-source nature of the work (implied by the project page) encourages further research in efficient, end-to-end V2A generation. [Flowley introduces a novel end-to-end flow-based architecture with Progressive Soft-masked Cross-Attention for precise video-to-audio generation, achieving state-of-the-art performance on VGGSound and outperforming larger closed-source models in zero-shot settings when augmented with its SoundCap captioning pipeline.] This paper presents a significant advancement in V2A generation by eliminating the need for multi-stage training and external alignment modules, offering a more efficient and effective solution for synthesizing temporally synchronized and semantically consistent audio from video.
Author of this paper? Send a correction.