SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrays
Abstract
Embodied, ego-centric intelligence fundamentally requires the ability to comprehend spatial audio within complex environments. While large audio-language models excel at mono-channel reasoning, they lack spatial awareness, discarding critical spatial cues that enable sound localization and that can improve the disentanglement of overlapping sound sources. To address this, we present SEA-LM, a Spatial Audio Understanding model. First, we introduce FOACODER, a layout-flexible spatial audio encoder trained on source localization and ego-centric voice activity detection objectives to encode First Order Ambisonics derived from variable-count, variable-position smart-glasses arrays via beamforming. We train a Multimodal Large Language Model (MLLM) to understand these spatial audio embeddings through a two-stage curriculum spanning six tasks, including sound localization and spatially selective transcription in settings with multiple speakers and overlapping sounds. To prevent the transcription outputs from dominating the next token prediction loss and overwhelming the direction predictions, we introduce a Spatio-temporal Weighted Cross-Entropy Loss. On our evaluation set, SEA-LM achieves lower azimuth and elevation MAE, higher temporal IoU, lower external-source hallucination and missing-source rates, and lower WER on most transcription tasks than compared baselines, while remaining robust across 1,211 smart-glasses array configurations with 4 to 9 microphones.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces SEA-LM, a spatial audio understanding model that integrates a layout-flexible FOA encoder with an MLLM to achieve robust ego-centric sound localization and transcription across diverse smart-glasses microphone arrays, addressing token dilution with a novel weighted loss function.
Methodology
The paper proposes SEA-LM, a framework for ego-centric spatial audio understanding. The core methodological contribution is the integration of a layout-flexible spatial audio encoder (FOACODER) with a Multimodal Large Language Model (MLLM). FOACODER is trained on First-Order Ambisonics (FOA) derived from variable-count smart-glasses microphone arrays via Ambisonic Signal Matching (ASM) beamforming. The encoder is pre-trained on joint Voice Activity Detection (VAD) and Sound Event Localization and Detection (SELD) objectives. The MLLM utilizes a dual-pathway architecture, combining frozen spatial embeddings from FOACODER with native monaural audio embeddings from a pretrained audio tower (Gemma-based). A key technical innovation is the Spatio-temporal Weighted Cross-Entropy Loss, which addresses the "token dilution" problem where dense transcription tokens overwhelm sparse spatial/temporal tokens during supervised fine-tuning. The data generation pipeline is rigorous, simulating realistic head and device scattering using COMSOL Multiphysics and integrating these Array Transfer Functions (ATFs) into room impulse response (RIR) simulations.
Results & evaluation
The evaluation is extensive, covering six distinct tasks ranging from holistic sound localization to targeted external transcription. The model is tested on 1,211 different microphone array configurations (4-9 mics) to demonstrate robustness. Results show significant improvements over baselines (Vanilla, Fine-tuned Mono, SELDNet+) in azimuth/elevation MAE, temporal IoU, and WER. The paper includes cohort analyses stratified by source count, overlap, and microphone count. However, the evaluation is entirely synthetic, which limits the direct applicability of the results to real-world noisy environments.
Reproducibility
The paper provides detailed descriptions of the architecture, loss functions, and data generation pipeline. It specifies hyperparameters, training steps, and hardware used. However, no code or model weights are released (no GitHub link provided), and the reliance on proprietary CAD models and specific simulation tools (COMSOL) may hinder full reproduction by external researchers.
Limitations
The primary limitation is the reliance on synthetic data for both training and evaluation. While the simulation is sophisticated, it may not capture all real-world artifacts (e.g., wind noise, non-stationary sources, complex reverberation beyond shoebox rooms). The model is also limited to stationary sources in the current evaluation. The generalization to other device form factors (phones, robots) is claimed but not empirically validated in the main results.
Broader impact
This work has significant implications for embodied AI, smart glasses, and assistive technologies. By enabling machines to understand *where* sounds are coming from and *who* is speaking (wearer vs. bystander), it enhances human-machine interaction in complex acoustic environments. The focus on ego-centric perspective is a crucial step toward practical wearable AI. The paper introduces SEA-LM, a spatial audio understanding model that integrates a layout-flexible FOA encoder with an MLLM to achieve robust ego-centric sound localization and transcription across diverse smart-glasses microphone arrays, addressing token dilution with a novel weighted loss function.
Author of this paper? Send a correction.