Here the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation
Abstract
Recent joint video-audio generation models have achieved strong semantic correspondence and temporal synchronization. However, applications such as AR/VR and interactive gaming further require stereo audio to provide an immersive sense, which remains largely overlooked. Effective stereo audio requires the perceived sound location to evolve consistently with the motion of its corresponding visual source. We refer to this property as Dynamic Spatial Correspondence and propose StereoBind, a framework that binds visual source motion to stereo sound generation. StereoBind uses motion tracks to coordinate visual motion and stereo audio through three complementary mechanisms. Visual Motion Binding establishes source-aware audiovisual correspondence, the Spatial Track Encoder captures absolute source positions, and Residual Track RoPE models relative motion. For supervision and evaluation, we construct StereoWorld-29K, a large-scale stereo audio-video dataset with paired motion tracks, and StereoWorldBench for measuring audiovisual spatial consistency. Experiments show that StereoBind substantially improves spatial alignment in stereo audio generation over existing models while preserving overall audiovisual quality.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces StereoBind, a framework for joint video-audio generation that enforces dynamic spatial correspondence by binding visual source motion to stereo audio generation through motion tracks, accompanied by a new large-scale dataset (StereoWorld-29K) and benchmark (StereoWorldBench). The technical contribution is strong, offering a novel conditioning mechanism (RT-RoPE) and a comprehensive evaluation suite for spatial audio fidelity, although the reliance on complex LLM-based data synthesis limits immediate reproducibility.
Methodology
The paper proposes StereoBind, a framework for joint video-audio generation that explicitly models "Dynamic Spatial Correspondence." The methodology is technically sound and well-structured, introducing three distinct mechanisms: Visual Motion Binding (VMB) for cross-modal source awareness, Spatial Track Encoder (STE) for absolute position conditioning, and Residual Track RoPE (RT-RoPE) for relative motion modeling. The integration of motion tracks as a conditioning signal for both video and audio branches is a logical and effective approach. The use of RoPE for spatial encoding is a clever adaptation of positional encoding techniques. However, the reliance on a specific backbone (LTX-2.3) and the complexity of the data pipeline (using multiple LLMs like Qwen3.5 and GPT-5.6 for data synthesis) adds significant engineering overhead. The method does not explicitly model ITDs/ILDs but relies on the model to learn these cues from the conditioned tracks, which is a valid but indirect approach.
Results & evaluation
The experimental evaluation is comprehensive. The authors construct a new dataset, StereoWorld-29K, which is a significant contribution, combining real-world data with synthetic data generated via a complex pipeline. The introduction of StereoWorldBench (SWBench) with specific metrics for spatial fidelity (ILD-W, SELD-Acc, etc.) is highly valuable for the field. The ablation studies clearly demonstrate the contribution of each component (VMB, STE, RT-RoPE). The human perceptual study, while small (25 volunteers), provides qualitative validation of the immersive experience. The comparison against strong baselines like Ovi and LTX-2.5 shows consistent improvements in spatial metrics without sacrificing general audio/video quality.
Reproducibility
Reproducibility is moderate. While the paper provides detailed descriptions of the architecture and data pipeline, the heavy reliance on proprietary or large-scale LLMs (Qwen3.5, GPT-5.6) for data synthesis makes exact reproduction difficult for the average researcher. The training setup (8 H20 GPUs, 2 days) is feasible for well-resourced labs. The code and dataset are not explicitly linked in the provided text, which is a minor drawback for immediate reproducibility.
Limitations
The primary limitation is the complexity and cost of the data generation pipeline, which relies on multiple large language models. The method is tested primarily on single-source scenarios; the paper mentions future work on multiple sources but does not address it. The evaluation metrics, while novel, are specific to this framework's definition of spatial consistency and may not generalize to all types of stereo audio. The human study is relatively small in scale.
Broader impact
This work has significant potential impact on AR/VR and interactive gaming applications, where immersive spatial audio is critical. By providing a framework and benchmark for dynamic spatial correspondence, it sets a new standard for evaluating audio-visual generation models. The dataset and benchmark will likely be adopted by the community to evaluate future models. The approach could be extended to other modalities or more complex 3D scenes. The paper introduces StereoBind, a framework for joint video-audio generation that enforces dynamic spatial correspondence by binding visual source motion to stereo audio generation through motion tracks, accompanied by a new large-scale dataset (StereoWorld-29K) and benchmark (StereoWorldBench). The technical contribution is strong, offering a novel conditioning mechanism (RT-RoPE) and a comprehensive evaluation suite for spatial audio fidelity, although the reliance on complex LLM-based data synthesis limits immediate reproducibility.
Author of this paper? Send a correction.