MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness
Abstract
Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MiDashengLM-Spatial unifies general audio understanding and spatial awareness in a single open-source model. The paper presents a rigorous integration of spatial and semantic encoders via hierarchical conditioning, supported by a large-scale synthetic data pipeline and comprehensive evaluations that demonstrate the model's ability to perform spatial reasoning without compromising general audio capabilities.
Methodology
The paper proposes MiDashengLM-Spatial, a unified architecture that integrates a monaural semantic encoder (Dasheng) and a spatial encoder (Spatial-Dasheng) via a Hierarchical Semantic-to-Spatial Conditioning (HSSC) module. The HSSC mechanism is a key architectural contribution, injecting intermediate semantic features into the spatial branch at multiple depths to facilitate the association of spatial cues with semantic content, particularly for overlapping sources. The spatial encoder itself is built on a Vision Transformer (ViT) backbone pre-trained with a Masked Autoencoder (MAE) objective, adapted for multi-channel input via a lightweight CNN fusion module. The training paradigm uses a multi-ACCDOA output format with Permutation-Invariant Training (PIT) to handle overlapping sources. The data pipeline is robust, synthesizing 1 million binaural clips and 6 million QA pairs using room impulse responses (RIRs) and LLM-generated scene descriptions, ensuring ground-truth alignment between spatial metadata and language.
Results & evaluation
The experimental evaluation is comprehensive. Spatial-Dasheng is evaluated on standard SELD benchmarks (MRSSound, EasyCom, STARSS23), showing strong performance and sim-to-real generalization. MiDashengLM-Spatial is evaluated on spatial reasoning benchmarks (MMAU-Pro, STAR-Bench) and general audio benchmarks (ASR, Captioning, QA). A notable strength is the "channel-swap consistency" evaluation, which rigorously tests whether the model's spatial reasoning is grounded in binaural cues rather than language priors or monaural artifacts. The model demonstrates that spatial awareness can be added without degrading general audio understanding, maintaining competitiveness with state-of-the-art 8B-scale LALMs.
Reproducibility
The paper provides high reproducibility by releasing source code and model checkpoints on GitHub and Hugging Face. Detailed implementation specifics, including data synthesis parameters, training curricula, and prompt templates for data generation, are provided in the appendices. The use of standard libraries like Pyroomacoustics and public datasets (LibriSpeech, AudioCaps) further enhances reproducibility.
Limitations
The model relies heavily on synthetic data for spatial supervision, which may limit its performance on real-world spatial audio with complex, unmodeled acoustic phenomena (e.g., non-stationary noise, irregular room geometries). The spatial encoder is limited to binaural input, and the maximum number of overlapping sources in training is capped at 3, which may not cover all real-world scenarios. Additionally, the model's performance on fine-grained spatial details in complex scenes remains less effective compared to coarse-grained attributes.
Broader impact
This work bridges the gap between general audio understanding and spatial perception, offering a unified framework that can be applied to immersive audio applications, virtual reality, and assistive technologies for the visually impaired. The open-source release of the model and data pipeline facilitates further research in spatial audio-language modeling and encourages the development of more human-like auditory perception systems. MiDashengLM-Spatial unifies general audio understanding and spatial awareness in a single open-source model. The paper presents a rigorous integration of spatial and semantic encoders via hierarchical conditioning, supported by a large-scale synthetic data pipeline and comprehensive evaluations that demonstrate the model's ability to perform spatial reasoning without compromising general audio capabilities.
Author of this paper? Send a correction.