HelixWorld: A Real-time Interactive Audio-Visual World Model
Abstract
World simulation is inherently multisensory, demanding synchronized visual and acoustic dynamics in real time. Yet prevailing interactive world models remain strictly silent, focusing exclusively on visual rendering and control while overlooking the acoustic dimension. We present HelixWorld, a real-time interactive audio-visual world model where visual scenes and camera-grounded spatial stereo sound co-evolve natively under user interaction. We curate a high-fidelity spatial audio-visual dataset with true stereo acoustics and metric camera poses, upon which we pre-train a bidirectional teacher conditioned on 6-DoF camera trajectories and user actions. To enable low-latency causal interaction, we distill the teacher into a few-step streaming student via an online trajectory distillation loss, sustaining drift-free joint audio-visual rollouts at 24 FPS on a single GPU. Furthermore, we formalize spatial-acoustic consistency and introduce HelixBench to evaluate whether synthesized sound fields faithfully track dynamic viewpoint motion. Extensive experiments demonstrate that HelixWorld matches state-of-the-art silent world models in visual fidelity and responsiveness, while significantly surpassing existing baselines in camera-aligned spatial-acoustic immersion.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
HelixWorld introduces a real-time interactive audio-visual world model that natively unifies 6-DoF camera control with spatial stereo sound generation, addressing the critical gap of acoustic realism in interactive simulations. The paper makes substantial contributions through a novel causal distillation framework that prevents multimodal drift in long-horizon rollouts, a rigorous data curation pipeline for high-fidelity spatial audio, and the introduction of HelixBench, a benchmark specifically designed to evaluate spatial-acoustic consistency in interactive world models.
Methodology
The paper proposes HelixWorld, a real-time interactive audio-visual world model. The core methodological contribution is a two-stage training pipeline: (1) Bidirectional teacher training using Projective Relative Positional Encoding (PRoPE) for 6-DoF camera conditioning and AdaLN-Zero for discrete action conditioning, coupled with cross-modal attention to bind visual geometry to acoustic features; and (2) Causal distillation into a few-step streaming student. The key innovation in distillation is the "online trajectory distillation loss," which anchors student rollouts to the teacher's Probability Flow ODE targets to mitigate the catastrophic multimodal drift and mode collapse typically observed in standard Distribution Matching Distillation (DMD) for long-horizon autoregressive generation. The data pipeline is also significant, employing a coarse-to-fine curation strategy that uses inter-channel energy probes to filter pseudo-stereo audio and multimodal LLMs to remove non-diegetic sound, resulting in a high-fidelity spatial audio-visual corpus.
Results & evaluation
The experimental evaluation is robust. The authors introduce HelixBench, a new benchmark of 1,015 human-verified cases specifically designed to evaluate spatial-acoustic consistency, a metric largely ignored by existing benchmarks. They demonstrate that HelixWorld outperforms state-of-the-art silent world models in visual fidelity and significantly surpasses baselines (including EchoWM and cascaded video-to-audio models) in spatial-acoustic immersion. Ablation studies effectively validate the necessity of progressive condition injection, the specific captioning format (three-track decoupled captions), and the online trajectory distillation loss for maintaining diversity and stability in long-horizon rollouts. The model achieves 24 FPS on a single H800 GPU, confirming real-time capability.
Reproducibility
The paper provides substantial implementation details, including specific hyperparameters for the distillation process, the structure of the temporal blocks, and the thresholds for data curation (e.g., inter-channel energy difference, MAD for static scenes). The use of open-source components (LTX-2.3, VGGT-Omega, Depth Anything V3) aids reproducibility. However, the specific code and the curated dataset are not explicitly linked in the provided text, and the "Unknown" institution status makes it harder to verify the team's prior work or access potential code repositories.
Limitations
The primary limitation is the reliance on a specific, potentially proprietary or not fully released dataset of 3.0k hours of curated spatial audio-visual data. The "Unknown" institution affiliation hinders assessment of the research group's standing and potential for code release. Additionally, while the model achieves 24 FPS, the computational cost of the bidirectional teacher training and the data curation pipeline (involving MLLMs and pose estimation) is likely high, limiting immediate adoption by smaller groups. The spatial audio evaluation relies on stereo panning correlation, which may not fully capture complex 3D sound fields (e.g., height or rear channels) beyond standard stereo.
Broader impact
This work has significant potential impact on the fields of interactive media, virtual reality, and game development. By natively coupling visual dynamics with spatially accurate audio under user control, it moves world models closer to true physical simulation. The introduction of HelixBench provides a necessary standard for evaluating the acoustic realism of interactive generative models, which has been a blind spot in the field. The techniques for mitigating drift in causal audio-visual distillation are broadly applicable to other multimodal streaming generation tasks. HelixWorld introduces a real-time interactive audio-visual world model that natively unifies 6-DoF camera control with spatial stereo sound generation, addressing the critical gap of acoustic realism in interactive simulations. The paper makes substantial contributions through a novel causal distillation framework that prevents multimodal drift in long-horizon rollouts, a rigorous data curation pipeline for high-fidelity spatial audio, and the introduction of HelixBench, a benchmark specifically designed to evaluate spatial-acoustic consistency in interactive world models.
Author of this paper? Send a correction.