WorldSonus: Bringing Sound to Worlds
Abstract
Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion. To address these demands, we introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41. For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. For spatial alignment, we leverage high-quality stereo supervision curated from diverse stereo and ambisonic data. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment. Project page: https://noizai.github.io/WorldSonus/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
WorldSonus introduces a modular, real-time causal video-to-audio framework that achieves state-of-the-art spatial and acoustic quality in interactive world models through two-timescale visual conditioning and training-only temporal alignment distillation.
Methodology
The paper proposes WorldSonus, a modular video-to-audio framework specifically designed for real-time, interactive world models. The core architectural contribution is a streaming causal autoregressive diffusion model that generates 100ms audio chunks using a bounded Ring-KV cache, ensuring constant memory usage. A key methodological innovation is the "two-timescale visual conditioning," which decouples long-horizon semantic continuity (handled by the AR backbone via chunk-level summaries) from fine-grained spatial/temporal alignment (handled by the flow head via direct frame-aligned DINOv3 features). Additionally, the paper introduces "ShiftNCE," a training-only distillation objective that uses a frozen Synchformer teacher to enforce temporal synchronization without adding inference latency. The system supports mid-stream prompt updates via in-place cross-attention cache replacement, enabling dynamic control over sound events without resetting the generation state.
Results & evaluation
The experimental evaluation is rigorous and comprehensive. The authors benchmark against state-of-the-art bidirectional models (AudioX, ThinkSound, PrismAudio) and a streaming mono baseline (V-AURA) on both open-domain (VGGSound) and interactive (gameplay/real-world) datasets. WorldSonus achieves competitive or superior acoustic quality (FAD, FD) and significantly better spatial alignment (BiasSkill) despite operating causally. The paper includes a robust ablation study isolating the contributions of ShiftNCE, visual feature representation, and chunk granularity. Notably, the authors conduct a paired counterfactual intervention to prove that text control is effective and not merely a result of visual transitions, and they perform a long-horizon stability test showing no drift over 30-second streams. A subjective user study further confirms human preference for WorldSonus over baselines.
Reproducibility
The paper provides extensive details in the appendices, including training hyperparameters, data curation pipelines, and metric specifications. The use of standard datasets (VGGSound, AudioSet) and public baselines facilitates comparison. However, the specific "Interactive" benchmark data is proprietary or newly collected, which may limit full reproduction of those specific results. The reliance on a frozen causal VAE from SoundReactor (which is noted to have lower reconstruction quality than non-causal counterparts) is a potential bottleneck, but the weights for the main model are not explicitly stated as released, though the project page is provided.
Limitations
The primary limitation is the reliance on a frozen causal audio VAE from SoundReactor, which the authors acknowledge has lower reconstruction quality than non-causal codecs. This caps the upper bound of audio fidelity. Additionally, high-quality stereo data is scarce, and the authors note that expanding authentic stereo data is a future goal. The model is evaluated on 48kHz audio, but the causal VAE's limitations may affect high-frequency detail.
Broader impact
This work is highly significant for the development of interactive world models, which are a key frontier in AI. By providing a modular, real-time, and controllable audio synthesis module, WorldSonus enables the creation of immersive, silent-no-more virtual environments. The technique of causal streaming with bounded memory is applicable to other real-time multimodal generation tasks. The rigorous evaluation of spatial alignment and interactive control sets a new standard for video-to-audio research. WorldSonus introduces a modular, real-time causal video-to-audio framework that achieves state-of-the-art spatial and acoustic quality in interactive world models through two-timescale visual conditioning and training-only temporal alignment distillation.
Author of this paper? Send a correction.