SonoWorld4D: From One Video to a 4D Audio-Visual Scene via Diffusion Synchronization
Abstract
Recent progress in visual scene generation has extended beyond static 3D scenes toward the more challenging problem of generating dynamic 4D scenes from minimal input. However, sound, despite being intrinsically coupled with dynamic objects and events, remains largely unmodeled in prior work. We introduce Video-to-4DAVScene, a new task that turns a video into an explorable, time-varying 4D audio-visual scene, and present the SonoWorld4D framework for this task. Given a monocular video, our method generates a 360 static background, reconstructs explicit motion trajectories of dynamic, sound-emitting foreground objects, and synthesizes a physically-grounded sound field consistent across multiple sources through dual-space audio diffusion synchronization. The resulting 4D scene supports real-time novel-view visual and spatial audio rendering. Evaluation on real-world dynamic scenes and a user study demonstrates SonoWorld4D generates 4D audio-visual scenes with perceptually realistic, physically grounded spatial audio consistent with scene geometry and visual content.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel task of Video-to-4DAVScene and a framework that effectively synchronizes visual dynamics with spatial audio using dual-space diffusion. The technical approach is rigorous, combining explicit 4D geometry with physically grounded acoustic rendering, and the introduction of a new dataset and evaluation protocol significantly advances the field of dynamic audio-visual scene generation.
Methodology
The paper proposes SonoWorld4D, a framework for generating explorable 4D audio-visual scenes from a single monocular video. The methodology is technically sophisticated, relying on a decomposed 4D visual representation (static background Gaussians + dynamic foreground objects with explicit trajectories) and a novel "dual-space diffusion synchronization" mechanism. This synchronization couples camera-space audio generation with object-space audio generation via forward and inverse acoustic rendering operators during the diffusion sampling process. The use of flow-matching and the specific bidirectional consistency constraints (rendering object sounds to camera space and separating/inverting camera sounds to object space) represent a strong technical contribution to the field of spatial audio generation. The integration of VLM-derived prompts for acoustic context is also a relevant modern addition.
Results & evaluation
The authors introduce a new dataset (SonoScene4D) with 54 clips of dynamic scenarios featuring moving sound sources, recorded with synchronized FOA audio at novel listener poses. This is a valuable contribution to the field, as existing datasets often lack dynamic spatial audio ground truth. The evaluation includes objective metrics (DoA errors, CC, AUC, CLAP) and a user study with 50 participants. The results show significant improvements over baselines like Sonic4D and OmniAudio in both spatial accuracy and semantic consistency. The qualitative analysis of Doppler effects and novel listener adaptation is convincing.
Reproducibility
The paper provides detailed descriptions of the pipeline, including the specific models used for depth estimation, segmentation, and Gaussian generation. However, the reliance on several proprietary or very recent models (e.g., HYWorld 2.0, Marble, SAM3, Kimi VLM) may limit immediate reproducibility for the community. The code is not explicitly linked in the provided text (only a demo page), which is a minor drawback for open science, though the demo page likely hosts assets.
Limitations
The method assumes outdoor scenes with limited reverberation, omitting reflected paths and complex room acoustics, which limits its applicability to indoor environments. The pipeline is complex and relies on a chain of foundation models, which may introduce error propagation. The dataset size (54 clips) is relatively small for robust statistical validation, though it is a new task. The computational cost of the dual-space diffusion synchronization and 4D Gaussian rendering is likely high, though real-time rendering is claimed for the final scene.
Broader impact
This work bridges the gap between 4D visual generation and spatial audio synthesis, enabling the creation of immersive, interactive worlds from minimal input. It has potential applications in virtual reality, game asset generation, and film pre-visualization. The introduction of a benchmark for dynamic spatial audio generation is particularly impactful for the research community. The paper introduces a novel task of Video-to-4DAVScene and a framework that effectively synchronizes visual dynamics with spatial audio using dual-space diffusion. The technical approach is rigorous, combining explicit 4D geometry with physically grounded acoustic rendering, and the introduction of a new dataset and evaluation protocol significantly advances the field of dynamic audio-visual scene generation.
Author of this paper? Send a correction.