LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds
Abstract
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
LiveFoley introduces a streaming causal autoregressive diffusion architecture that achieves imperceptible latency and high-quality audio-visual alignment by decoupling frontend look-ahead from receptive field size via a Trailing Sliding Window and enhancing backend denoising with historical and future-warmup contexts. The technical contribution is significant for the field of real-time multimodal generation, offering a practical solution to the latency-quality trade-off that has hindered the adoption of high-fidelity V2A models in interactive environments.
Methodology
The paper proposes LiveFoley, a streaming video-to-audio generation system that addresses the latency-quality trade-off in real-time applications. The core contribution is the "Trailing Sliding Window" mechanism, which allows non-causal segment-level encoders (like Synchformer) to operate in a streaming context without retraining by anchoring the window tail to the current frame and using first-frame padding. This decouples look-ahead latency from the receptive field size. Additionally, the "Unified Varhop Framework" introduces a Synchronizer module to handle variable hop sizes, allowing the system to adapt to different latency budgets dynamically. For the audio backend, the authors modify the Local Diffusion Transformer (LocDiT) by introducing "History LocDiT" (using past context) and "Future-Warmup LocDiT" (using approximate future conditions) to improve generation quality compared to isolated patch denoising. The methodology is sound and logically addresses the identified bottlenecks in existing streaming V2A systems.
Results & evaluation
The paper includes a user study to quantify human perceptual thresholds for audio-visual delay, which is a valuable contribution to the field. Experiments demonstrate that LiveFoley matches offline models in temporal alignment while maintaining low latency. It claims to lead streaming systems in generation quality. The use of a single A800 GPU for real-time performance is a strong practical result. However, the text provided is truncated, so the full extent of quantitative comparisons (e.g., FAD, CLAP scores) against specific baselines like V-AURA or SoundReactor is not fully visible, though the abstract and intro suggest comprehensive testing.
Reproducibility
The paper provides architectural details and a project page. The use of standard components (MiniCPM, Synchformer, flow-matching) aids reproducibility. The specific implementation details of the Synchronizer and the training protocol for the variable hop sizes are described, but code availability is not explicitly confirmed in the provided text (only a project page URL is given).
Limitations
The system relies on a causal LM backbone which may still struggle with very long-range dependencies compared to global bidirectional models. The "Future-Warmup" mechanism introduces a small amount of additional latency (0.08s) which, while below perceptual thresholds, adds complexity. The evaluation is primarily focused on latency and alignment; subjective audio quality metrics (MOS) are likely included in the full paper but are not detailed in the truncated text.
Broader impact
This work has significant potential for immersive interactive applications such as AR/VR, video games, and live streaming, where real-time audio feedback is crucial. By providing a plug-and-play sound engine that works with arbitrary visual pipelines, it lowers the barrier for integrating high-quality synchronized audio into interactive media. LiveFoley introduces a streaming causal autoregressive diffusion architecture that achieves imperceptible latency and high-quality audio-visual alignment by decoupling frontend look-ahead from receptive field size via a Trailing Sliding Window and enhancing backend denoising with historical and future-warmup contexts. The technical contribution is significant for the field of real-time multimodal generation, offering a practical solution to the latency-quality trade-off that has hindered the adoption of high-fidelity V2A models in interactive environments.
Author of this paper? Send a correction.