Spatial Transfusion: From Structured Spatial Scene Understanding to Generation and Editing
Abstract
Spatial audio conveys not only what sounds are present, but where they originate and how they interact with the environment. Recent audio-language models increasingly understand and reason about such scenes, yet spatial audio understanding, generation, and editing remain largely separate. We present Spatial Transfusion, a unified first-order ambisonics (FOA) framework that extends a pretrained spatial audio understanding model to generation and editing through flow matching over VAE latents. We formulate editing as a structured scene transformation over a taxonomy of atomic edits. Keeping the pretrained backbone frozen, we introduce modality-based LoRA routing and a curriculum that progressively adds generation, editing, and natural-language interaction. Experiments show that Spatial Transfusion largely preserves spatial understanding while outperforming full fine-tuning in generation and editing. The resulting model supports controllable FOA generation and editing through both structured and natural-language interfaces. Audio samples are available at https://microsoft.github.io/spatial-transfusion/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Spatial Transfusion introduces a unified framework for FOA audio understanding, generation, and editing by extending a pretrained spatial audio language model with flow matching and modality-based LoRA routing. The paper demonstrates that this approach preserves spatial understanding while enabling high-quality, controllable generation and editing, outperforming full fine-tuning baselines and establishing a new paradigm for spatial audio manipulation.
Methodology
The paper proposes Spatial Transfusion, a unified framework for First-Order Ambisonics (FOA) audio understanding, generation, and editing. The core methodological contribution is the adaptation of a pretrained spatial audio language model (Sci-Phi) using a "Transfusion" paradigm, which combines autoregressive text modeling with flow-matching for continuous audio latents. A key technical innovation is the use of modality-based LoRA routing, where separate LoRA adapters are applied to audio and text tokens, allowing the frozen backbone to learn generative capabilities without catastrophic forgetting of understanding tasks. The authors define a structured taxonomy of atomic edits (semantic, acoustic, spatial, temporal) and employ a four-stage curriculum to progressively introduce generation and editing capabilities. The approach effectively leverages the structured nature of spatial scenes to enable precise control over generation and editing, distinguishing it from generic audio generation models.
Results & evaluation
The experimental setup is rigorous, utilizing a held-out set from the FOA-MEIR dataset with real room impulse responses. The evaluation metrics are comprehensive, covering spatial fidelity (FAD-PSELD, FOA covariance), semantic alignment (CLAP), and editing-specific metrics (EditFLAM). The results demonstrate that the proposed LoRA routing strategy outperforms full fine-tuning in generation and editing while preserving understanding capabilities. Ablation studies on routing strategies (modality-based vs. task-based vs. dense) provide strong evidence for the effectiveness of the chosen approach. The inclusion of natural language instruction tuning (Stage 4) further validates the flexibility of the framework.
Reproducibility
The paper provides detailed descriptions of the architecture, training curriculum, and hyperparameters. However, specific details on the FOA-VAE implementation and the exact simulation pipeline parameters are referenced to prior work (Sci-Phi, FOA-VAE), which may limit immediate reproducibility without access to those codebases. The availability of a demo page with audio samples is a positive indicator, but a full code release is not explicitly mentioned in the text provided.
Limitations
The model is trained primarily on simulated data, which may limit its generalization to real-world noisy environments, although the evaluation uses real RIRs. The framework is specific to FOA, and extension to higher-order ambisonics or binaural audio is not explored. The editing operations are limited to the defined atomic taxonomy, and complex, multi-step edits may not be supported. The reliance on a specific pretrained backbone (Phi-4-MM) may constrain the model's scalability.
Broader impact
This work bridges the gap between spatial audio understanding and generation, offering a unified interface for immersive audio creation. The structured editing capabilities have potential applications in virtual reality, gaming, and film production, where precise control over soundscapes is crucial. The framework's ability to handle both structured JSON and natural language inputs makes it accessible to both technical and non-technical users. Spatial Transfusion introduces a unified framework for FOA audio understanding, generation, and editing by extending a pretrained spatial audio language model with flow matching and modality-based LoRA routing. The paper demonstrates that this approach preserves spatial understanding while enabling high-quality, controllable generation and editing, outperforming full fine-tuning baselines and establishing a new paradigm for spatial audio manipulation.
Author of this paper? Send a correction.