Visually-Guided Spatial Audio Generation for $360^\circ$ In-the-Wild Speech Scenes
Abstract
Spatial audio is a key component of immersive $360^\circ$ media, yet high-quality spatial capture remains limited in real-world speech-dominant scenes. We study visually guided First-Order Ambisonics (FOA) speech spatialization in the wild: given aligned $360^\circ$ video and an omnidirectional audio track, we recover the missing directional FOA components. To support this task, we introduce YT-SPEECH, a speech-oriented $360^\circ$ video-FOA dataset curated from YouTube. We propose a two-stage Localizer-Renderer framework, where an audio-visual segmentation backbone provides frame-wise spatial heatmaps and a conditional complex-domain U-Net reconstructs directional FOA signals from the omnidirectional channel. A confidence-based gating strategy stabilizes conditioning under ambiguous acoustic conditions. Experiments show improved reconstruction fidelity, spatial accuracy, and perceptual speech quality relative to ablated variants and prior approaches.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a practical framework for visually guided FOA speech spatialization, leveraging existing AVS models to guide complex-domain audio generation, and contributes a new dataset to the field.
Methodology
The paper proposes a two-stage framework, Localizer-Renderer, for visually guided First-Order Ambisonics (FOA) speech spatialization. The Localizer utilizes an Audio-Visual Segmentation (AVS) backbone to generate spatial heatmaps from 360-degree video, which are then used as conditional priors for a complex-domain U-Net Renderer. The approach introduces a confidence-based gating mechanism to modulate the conditioning strength based on the entropy and peakiness of the spatial prior. While the integration of AVS for spatial grounding in audio generation is a logical extension of current multimodal trends, the core methodology relies on established components (U-Net, AVS backbones, FOA encoding) rather than introducing fundamentally new architectural primitives. The "confidence gating" is a heuristic modification rather than a deep theoretical innovation. The use of complex-valued networks for phase preservation is a known best practice in this specific subfield, not a novel contribution.
Results & evaluation
The authors curate a new dataset, YT-SPEECH, which is a valuable resource for the community, addressing the scarcity of paired 360-video and FOA speech data. The experimental evaluation includes ablation studies on the Localizer and Renderer components, as well as comparisons with analytic baselines (AmbiEnc, Pyroomacoustics) and a prior end-to-end model (SAG). The results show improvements in spatial accuracy (DOA error) and perceptual quality (PESQ, MOS). However, the subjective evaluation is limited to a small-scale listening test with only 9 participants, which reduces the statistical power of the perceptual claims. The comparison with SAG is conducted on SAG's own datasets, which is good for compatibility, but the lack of comparison with other recent state-of-the-art video-to-audio models (e.g., ViSAGe, OmniAudio) limits the assessment of relative performance in the broader landscape.
Reproducibility
The paper provides sufficient detail on the architecture, loss functions, and training strategy (pretraining on Sphere360, fine-tuning on YT-SPEECH). The dataset curation pipeline is described in detail. However, the code is not explicitly linked in the text (only a demo URL is provided), and the specific hyperparameters for the AVS backbone fine-tuning are not fully enumerated, which may hinder exact reproduction. The reliance on proprietary or hard-to-access components (e.g., specific Whisper versions, YOLOv8-nano configurations) adds minor friction.
Limitations
The authors acknowledge the limited scale of the dataset (8.9 hours) and reduced stability under overlapping sources and acoustically complex scenes. The method's performance is heavily dependent on the quality of the visual localization; if the AVS model fails to segment the speaker correctly, the spatial prior will be misleading, potentially degrading performance. The small sample size of the subjective listening test is a significant limitation for validating perceptual claims.
Broader impact
This work contributes to the field of immersive media and spatial audio, which has applications in virtual reality, telepresence, and accessible media for visually impaired users. The release of the YT-SPEECH dataset is a positive contribution to the community, enabling further research in visually guided spatial audio. However, the use of YouTube data raises potential copyright and privacy concerns regarding the training data, although the authors note manual inspection for quality and consistency. This paper presents a practical framework for visually guided FOA speech spatialization, leveraging existing AVS models to guide complex-domain audio generation, and contributes a new dataset to the field.
Author of this paper? Send a correction.