Audible World Models: Spatially Aware Sound Generation for 3D Worlds
Abstract
Text- and image-conditioned world generators can create visually rich 3D environments, yet these worlds often remain silent or rely on soundtracks synthesized solely from text or rendered video. Although such audio can convey what should be heard, it lacks an explicit representation of where sound sources are located and how their perceived sound should vary with listener movement. We introduce Audible World Models, a training-free framework that incorporates sound into the generated world state. Starting from a text prompt, our system constructs a panoramic 3D proxy, separates it into semantic layers, identifies sound-producing foreground objects and ambient background regions, and synthesizes dry audio for each sound label. It then anchors these sources to reconstructed geometry and renders listener-dependent spatial audio using geometric acoustic propagation. By explicitly linking semantics, geometry, and sound propagation, the framework maintains persistent source locations while adapting the rendered audio to changes in listener viewpoint and motion. Experiments across 80 generated scenes demonstrate substantial gains in spatial consistency over text-, video-, and panorama-conditioned baselines, while preserving competitive semantic alignment. VLM-based assessments and human evaluations further indicate that our soundtracks are preferred for their audio-visual consistency, spatial plausibility, and motion-dependent behavior.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a training-free agentic framework that integrates spatially aware sound generation into 3D world models by explicitly linking semantic labels, reconstructed geometry, and geometric acoustic propagation. This approach effectively solves the problem of spatial inconsistency in generated audio by grounding sound sources in a persistent 3D world state, resulting in significant improvements in spatial consistency and audio-visual alignment compared to existing video- and text-conditioned baselines.
Methodology
The paper proposes a training-free, agentic pipeline that bridges visual world generation and spatial audio synthesis. The core methodological contribution is the formulation of an "audible world state" that explicitly separates semantic sound labels, dry audio assets, 3D source placements, and acoustic parameters. By leveraging VLMs to parse panoramic 3D proxies into semantic layers (foreground objects vs. ambient background), the system grounds generated audio to reconstructed geometry. It then utilizes a geometric acoustic simulator (GSound) to render listener-dependent binaural audio. This approach is technically sound and effectively addresses the lack of spatial persistence in current video-to-audio models. The use of an agentic framework for parameterization and source grounding is a clever engineering solution that avoids the need for end-to-end training of complex multimodal models.
Results & evaluation
The evaluation is comprehensive, covering both semantic alignment and spatial consistency. The authors utilize 80 generated scenes and compare against strong baselines including Stable Audio 1.0, MMAudio, SEE-2-SOUND, and OmniAudio. The use of Direction of Arrival (DoA) MAE and F1 scores for spatial consistency is a rigorous metric that directly tests the core claim of the paper. The inclusion of human evaluations for preference and VLM-based assessments adds depth to the subjective quality assessment. The results show substantial gains in spatial consistency (6.91° DoA MAE for isolated sources) while maintaining competitive semantic scores, validating the effectiveness of the proposed framework.
Reproducibility
The paper provides a clear description of the pipeline stages and the specific tools used (HunyuanWorld, Stable Audio 1.0, GSound). However, as a training-free system relying on specific API calls and agentic prompts, full reproducibility depends on the stability of these third-party models and the specific prompt templates (referenced in appendices). The lack of a public code repository or demo link in the provided text slightly hinders immediate reproducibility, though the modular nature of the system makes it feasible to replicate with the cited components.
Limitations
The system relies on the quality of the initial panoramic generation and depth estimation; errors in the 3D proxy will propagate to source placement. The acoustic parameterization is heuristic and "conservative," meaning it may not capture complex material-specific acoustic properties accurately. Additionally, the pipeline is computationally intensive due to the multi-stage agentic process and geometric simulation, which may limit real-time applications. The evaluation is limited to 80 scenes, which, while sufficient for a proof of concept, is relatively small for a general claim about world models.
Broader impact
This work has significant implications for the development of immersive virtual environments, VR/AR applications, and interactive storytelling. By making audio a persistent, geometry-aware property of the world state rather than a post-hoc soundtrack, it paves the way for more consistent and engaging user experiences. The framework's modularity allows for easy integration of future improvements in audio generation or 3D reconstruction, making it a robust foundation for future research in multimodal world modeling. The paper introduces a training-free agentic framework that integrates spatially aware sound generation into 3D world models by explicitly linking semantic labels, reconstructed geometry, and geometric acoustic propagation. This approach effectively solves the problem of spatial inconsistency in generated audio by grounding sound sources in a persistent 3D world state, resulting in significant improvements in spatial consistency and audio-visual alignment compared to existing video- and text-conditioned baselines.
Author of this paper? Send a correction.