AURA: Auditory Spatial Intelligence via a Unified Representation from a General-Audio Encoder through Complete Alignment
Abstract
Answering spatial questions about multi-source scenes requires a model both to identify each sound and to locate it. Existing spatial audio-language models obtain spatial information from a dedicated encoder, which separates spatial and semantic processing into distinct feature spaces. We study whether a general-audio codec trained without spatial supervision can serve both purposes. A controlled layer-wise analysis shows that the frozen codec retains spatial information, with lateral cues carried by the difference between the two ear channels. This information, however, is not readily accessible: in dual-source scenes, a model trained on these features predicts left/right at chance and, on the remaining axes, assigns the detected direction to the queried and the non-queried source equally often. We attribute this failure to an audio--text information asymmetry, in which the audio contains both sources while the answer describes only the queried one. A controlled $2\times2$ experiment shows that either completing the scene description or exposing the inter-channel difference lifts left/right accuracy above chance, whereas binding directions to the queried source requires both. Building on these findings, AURA keeps the codec and the language model frozen and exceeds the strongest dedicated system on SpatialSoundQA by 10.31 points in dual-source direction accuracy, with about 40\% of its trainable parameters.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper demonstrates that a frozen general-audio codec can support spatial reasoning by addressing audio-text information asymmetry through explicit inter-channel features and comprehensive scene supervision, achieving superior dual-source localization performance with reduced parameter count.
Methodology
The paper proposes AURA, a framework for spatial audio understanding that leverages a frozen general-audio codec (X-Codec) rather than a dedicated spatial encoder. The methodology is grounded in a rigorous layer-wise analysis of the codec's representations, identifying that spatial cues (specifically lateral cues) are preserved in deeper RVQ layers and are best accessed via inter-channel differences. The core innovation is the "information asymmetry" hypothesis, which posits that standard QA training fails because the audio contains multiple sources while the text target describes only one. To address this, AURA introduces two components: (1) a representation-side intervention adding explicit inter-channel difference slots to the audio tokens, and (2) a text-side intervention using "scene supervision" where the model must describe all sources before answering. The approach is technically sound, effectively bridging the gap between general audio representations and specific spatial reasoning tasks without retraining the heavy backbone models.
Results & evaluation
The experiments are well-controlled, particularly the 2x2 ablation study that isolates the effects of scene supervision and inter-channel differences. The results on SpatialSoundQA show a significant improvement (10.31 points) over the strongest dedicated system in dual-source direction accuracy, which is a strong claim given the parameter efficiency (40% of trainable parameters). The evaluation correctly focuses on the challenging dual-source scenarios where existing models fail. However, the reliance on a single benchmark (SpatialSoundQA) and the lack of evaluation on other spatial datasets (like BiDepth or real-world recordings) slightly limits the generalizability of the findings.
Reproducibility
The paper provides detailed descriptions of the architecture modifications (6-slot representation, LoRA modules) and the training strategy. However, specific hyperparameters, learning rates, and exact implementation details for the "geometric supervision" prompt construction are not fully detailed in the provided text. The use of standard components (Qwen3-8B, X-Codec) aids reproducibility, but the specific prompt engineering for scene supervision is a critical component that requires careful replication.
Limitations
The primary limitation is the dependence on simulated binaural data (SpatialSoundQA) for training and evaluation. It is unclear how well the model generalizes to real-world binaural recordings with complex noise and reverberation. Additionally, the model is frozen, which means it cannot adapt to new spatial concepts or room types beyond what the codec has seen. The "scene supervision" approach increases the length of the model's output, which may have computational implications for real-time applications.
Broader impact
This work has significant implications for the development of efficient spatial audio models. By demonstrating that general audio codecs retain spatial information, it challenges the necessity of dedicated spatial encoders, potentially simplifying the architecture of future audio-language models. The insights into information asymmetry in multi-source scenes are applicable to other multimodal tasks where the input contains more information than the target output. This could influence how datasets are constructed and how models are trained for complex scene understanding. The paper demonstrates that a frozen general-audio codec can support spatial reasoning by addressing audio-text information asymmetry through explicit inter-channel features and comprehensive scene supervision, achieving superior dual-source localization performance with reduced parameter count.
Author of this paper? Send a correction.