AudioSphere: Towards Self-Supervised Spatial Audio Representation Models
Abstract
Recent audio embedding models learn representations that transfer across clip- and frame-level tasks, yet they operate almost exclusively on single-channel signals and are blind to spatial structure. Spatial audio models capture that structure but rely on clip-level supervision or contrastive objectives, yielding embeddings unsuitable for frame-level tasks like sound event detection and localization (SELD). We argue this dichotomy is not fundamental: patch-level spatial self-supervision can jointly learn acoustic and spatial features at both temporal granularities. We introduce AudioSphere, a multi-channel masked autoencoder trained on Ambisonics recordings that reconstructs both spectral content and directional cues via a joint spatiotemporal masking strategy that masks both time-frequency patches and intensity vectors. AudioSphere supports frame-level predictions while producing strong clip-level representations via aggregation. Alongside the model, we release RealSELD, a standardized benchmark for evaluating frozen backbones on real-world multi-channel SELD. AudioSphere enables, for the first time, SELD with a frozen self-supervised backbone, while matching or exceeding prior self-supervised audio embedding models on clip-level classification tasks spanning environmental sound, speech, and music, and on frame-level sound event detection. All code and data are available at https://github.com/labhamlet/AudioSphere and https://huggingface.co/labhamlet.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AudioSphere introduces a multi-channel masked autoencoder with joint spatiotemporal masking to learn self-supervised spatial audio representations, achieving state-of-the-art performance on both real-world SELD and monaural audio tasks. The paper makes a significant technical contribution by bridging the gap between monaural SSL and spatial audio modeling, demonstrating that spatial cues can be learned effectively without explicit supervision and that such representations transfer robustly to a wide range of downstream tasks. The introduction of the RealSELD benchmark and the open-sourcing of the model and data pipeline further enhance the paper's impact, providing the community with essential tools for advancing spatial audio research.
Methodology
The paper proposes AudioSphere, a multi-channel masked autoencoder (MAE) designed for self-supervised learning of spatial audio representations. The core innovation lies in the joint modeling of acoustic content (log-mel spectrograms) and spatial cues (intensity vectors) derived from First-Order Ambisonics (FOA). The authors introduce a "tube masking" strategy that masks time-frequency patches across all channels simultaneously, preventing the model from trivially copying information from unmasked channels at the same spatial location. This forces the encoder to learn correlations between acoustic features and directional cues. The architecture utilizes a ViT-Base encoder and a decoder with local-global attention. A significant methodological contribution is the data generation pipeline, which synthesizes 70,000 Ambisonic room impulse responses (ARIRs) using Matterport3D meshes and Monte Carlo ray tracing, paired with online mixing of AudioSet clips. This addresses the scarcity of large-scale, labeled spatial audio data. The use of Ambisonics Signal Matching (ASM) allows the model to generalize across different microphone array geometries (tetrahedral, planar, spherical) without retraining, a robust engineering choice that enhances utility.
Results & evaluation
The experimental evaluation is comprehensive, addressing both the transferability of spatial features to monaural tasks and the efficacy of the model for spatial tasks. The authors introduce RealSELD, a standardized benchmark for real-world SELD, which is a valuable contribution to the field. Results show that AudioSphere, when used as a frozen backbone with shallow MLP probes, outperforms fully supervised baselines on static SELD tasks and transfers competitively to dynamic environments. Crucially, the paper demonstrates that spatial pre-training does not degrade performance on monaural tasks; in fact, it matches or exceeds prior state-of-the-art monaural SSL models (like MW-MAE) on the HEAR benchmark suite, which includes environmental sound, speech, and music classification. The ablation studies on masking strategies (tube vs. per-channel) provide strong evidence for the necessity of the proposed joint masking approach. The comparison with MUSIC and Binaural Signal Matching pipelines on real-world data further validates the practical superiority of the learned representations.
Reproducibility
The authors provide high reproducibility by releasing all code, data, and model weights on GitHub and HuggingFace. The detailed description of the simulation pipeline, including the specific parameters for Matterport3D scenes and the online mixing process, allows other researchers to replicate the training data generation. The use of standard components (ViT, ASM) and clear architectural specifications further supports reproducibility. The release of the RealSELD benchmark kit is particularly beneficial for the community, enabling standardized evaluation of future spatial audio models.
Limitations
A primary limitation is the reliance on simulated room impulse responses for pre-training. While the authors argue that the model generalizes to real-world recordings, the gap between simulated and real-world acoustic conditions (e.g., non-linearities, complex material interactions) may still exist, though the results on RealSELD suggest this gap is manageable. The model is trained on First-Order Ambisonics, which limits the spatial resolution to 4 channels; higher-order Ambisonics could provide finer directional detail but would increase computational cost. Additionally, the pre-training corpus, while large (4.8k hours of AudioSet mixed with simulations), is still smaller than the massive corpora used for some monaural models (e.g., Dasheng's 272k hours), which might limit the ceiling of performance on certain acoustic tasks.
Broader impact
This work has significant implications for the field of spatial audio understanding. By demonstrating that self-supervised learning can effectively capture joint acoustic-spatial features, it opens the door to scalable, label-free training for spatial audio models. The release of RealSELD provides a much-needed standardized benchmark for evaluating real-world SELD, addressing a long-standing issue of fragmented evaluation in this subfield. The ability to use a frozen backbone for both monaural and spatial tasks suggests that spatial pre-training can be a general-purpose enhancement for audio representation learning, potentially benefiting applications in augmented reality, virtual reality, and assistive technologies for the deaf and hard of hearing. AudioSphere introduces a multi-channel masked autoencoder with joint spatiotemporal masking to learn self-supervised spatial audio representations, achieving state-of-the-art performance on both real-world SELD and monaural audio tasks. The paper makes a significant technical contribution by bridging the gap between monaural SSL and spatial audio modeling, demonstrating that spatial cues can be learned effectively without explicit supervision and that such representations transfer robustly to a wide range of downstream tasks. The introduction of the RealSELD benchmark and the open-sourcing of the model and data pipeline further enhance the paper's impact, providing the community with essential tools for advancing spatial audio research.
Author of this paper? Send a correction.