SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videos
Abstract
We propose SALSA-V, a multimodal video-to-audio generation model capable of synthesizing highly synchronized, high-fidelity long-form audio from silent video content. Our approach introduces a masked diffusion objective, enabling audio-conditioned generation and the seamless synthesis of audio sequences of unconstrained length. Additionally, by integrating a shortcut loss into our training process, we achieve rapid generation of high-quality audio samples in as few as eight sampling steps, paving the way for near-real-time applications without requiring dedicated fine-tuning or retraining. We demonstrate that SALSA-V significantly outperforms existing state-of-the-art methods in both audiovisual alignment and synchronization with video content in quantitative evaluation and a human listening study. Furthermore, our use of random masking during training enables our model to match spectral characteristics of reference audio samples, broadening its applicability to professional audio synthesis tasks such as Foley generation and sound design.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SALSA-V introduces a novel approach to video-to-audio generation that significantly enhances synchronization and audio quality while enabling rapid generation. The comprehensive evaluation of its capabilities and the potential for real-world applications underscore its importance in advancing the field of multimodal machine learning.
Methodology
The methodology presented in SALSA-V is innovative, leveraging a masked diffusion objective and shortcut loss to enable rapid audio generation from video. The integration of audio conditioning and long-form generation capabilities is a significant advancement over existing models. The use of contrastive learning for synchronization features is well-justified and effectively addresses the limitations of prior approaches. However, the complexity of the model architecture and training process may pose challenges for replication and understanding.
Results & evaluation
The experimental evaluation is thorough, utilizing multiple datasets and metrics to assess audio quality, synchronization, and human preference. The results demonstrate clear improvements over state-of-the-art methods, particularly in synchronization and audio fidelity. The human evaluation adds valuable qualitative insights, although the gap in subjective audio quality compared to MMAudio suggests room for further refinement.
Reproducibility
The paper provides detailed implementation details, including model architecture, training procedures, and evaluation metrics, which supports reproducibility. However, the lack of a publicly available code repository limits accessibility for other researchers looking to replicate or build upon this work.
Limitations
The paper acknowledges limitations, particularly regarding the subjective audio quality compared to larger models. The reliance on progressive outpainting for long-form generation may also introduce challenges in maintaining audio coherence across scene changes. Additionally, the model's performance may be constrained by its parameter count, suggesting that scaling could yield further improvements.
Broader impact
The potential applications of SALSA-V are significant, particularly in fields such as film production, gaming, and virtual reality, where high-quality audio generation is crucial. The ability to generate synchronized audio in near-real-time could revolutionize workflows in these industries, enhancing creative processes and reducing production times. SALSA-V introduces a novel approach to video-to-audio generation that significantly enhances synchronization and audio quality while enabling rapid generation. The comprehensive evaluation of its capabilities and the potential for real-world applications underscore its importance in advancing the field of multimodal machine learning.
Author of this paper? Send a correction.