SoundReactor: Frame-level Online Video-to-Audio Generation
Abstract
Prevailing Video-to-Audio (V2A) generation models operate offline, assuming an entire video sequence or chunks of frames are available beforehand. This critically limits their use in interactive applications such as live content creation and emerging generative world models. To address this gap, we introduce the novel task of frame-level online V2A generation, where a model autoregressively generates audio from video without access to future video frames. Furthermore, we propose SoundReactor, which, to the best of our knowledge, is the first simple yet effective framework explicitly tailored for this task. Our design enforces end-to-end causality and targets low per-frame latency with audio-visual synchronization. Our model's backbone is a decoder-only causal transformer over continuous audio latents. For vision conditioning, it leverages grid (patch) features extracted from the smallest variant of the DINOv2 vision encoder, which are aggregated into a single token per frame to maintain end-to-end causality and efficiency. The model is trained through a diffusion pre-training followed by consistency fine-tuning to accelerate the diffusion head decoding. On a benchmark of diverse gameplay videos from AAA titles, our model successfully generates semantically and temporally aligned, high-quality full-band stereo audio, validated by both objective and human evaluations. Furthermore, our model achieves low per-frame waveform-level latency (26.3ms with the head NFE=1, 31.5ms with NFE=4) on 30FPS, 480p videos using a single H100. Demo samples are available at https://koichi-saito-sony.github.io/soundreactor/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SoundReactor, a novel framework for online video-to-audio generation that operates frame-by-frame, enabling real-time audio synthesis with low latency and high-quality synchronization. This work represents a meaningful advancement in the field of audio-visual machine learning, with significant implications for interactive applications and content creation.
Methodology
The paper introduces a novel approach to frame-level online video-to-audio generation, which is a significant advancement over traditional offline methods. The use of a decoder-only causal transformer for audio generation, combined with grid features from the DINOv2 vision encoder, is innovative and effectively addresses the challenges of maintaining causality and low latency. The methodology is well-structured, with a clear focus on end-to-end causality and synchronization between audio and video. The training process, which involves diffusion pre-training followed by consistency fine-tuning, is a thoughtful approach to enhance model performance.
Results & evaluation
The experiments are robust, utilizing a benchmark of diverse gameplay videos from AAA titles, which adds credibility to the findings. The reported results demonstrate that the model generates high-quality audio that is semantically and temporally aligned with the video content. Objective and human evaluations further validate the effectiveness of the model, although the paper could benefit from more detailed comparisons with existing state-of-the-art methods to contextualize its performance.
Reproducibility
The paper provides a clear description of the model architecture and training procedure, which is essential for reproducibility. However, it lacks detailed information regarding the datasets used, including their size and specific characteristics, which could hinder full reproducibility by other researchers. The availability of demo samples is a positive aspect, but a code repository would further enhance reproducibility.
Limitations
One limitation is the reliance on a specific type of video content (gameplay videos), which may not generalize well to other video domains. Additionally, while the model achieves low per-frame latency, the computational requirements may limit its applicability in resource-constrained environments. The paper could also explore the model's performance on a wider variety of audio-visual content to assess its versatility.
Broader impact
The potential applications of this research are significant, particularly in interactive media, live content creation, and generative models for virtual environments. The ability to generate audio in real-time from video could revolutionize fields such as gaming, virtual reality, and content creation, making it easier for creators to produce immersive experiences. The implications for accessibility in media production are also noteworthy, as this technology could enable more inclusive content creation. The main contribution of this paper is the introduction of SoundReactor, a novel framework for online video-to-audio generation that operates frame-by-frame, enabling real-time audio synthesis with low latency and high-quality synchronization. This work represents a meaningful advancement in the field of audio-visual machine learning, with significant implications for interactive applications and content creation.
Author of this paper? Send a correction.