SwanWeave:One-Stage Multi-Task Instruction-Guided 3D Spatial Audio Editing
Abstract
Spatial audio editing modifies an existing soundfield according to a user's instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at https://swanaigc.github.io/#swanweave, code can be found at: https://github.com/MM-Speech/SwanWeave.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel one-stage multi-task framework for 3D spatial audio editing that effectively addresses the challenges of preserving unedited regions while executing complex compound instructions. By combining a dual-level Mixture-of-Experts architecture with preference optimization tailored for spatial edits, the method achieves state-of-the-art performance in both semantic and spatial fidelity, marking a significant step forward in controllable spatial audio generation.
Methodology
The paper proposes SwanWeave, a one-stage framework for instruction-guided 3D First-Order Ambisonic (FOA) spatial audio editing. The methodology is robust, combining a flow-matching latent editor with a novel Spatial Edit Mixture-of-Experts (SE-MoE) module. The SE-MoE design is particularly interesting, utilizing dual-level routing: task-level experts selected based on the instruction to handle compound edits, and frame-level routed/null experts to handle local temporal changes and preservation of unedited regions. The use of a "null expert" that outputs zero tensors for unchanged regions is a clever architectural choice to prevent drift in preserved audio. The training strategy is well-structured, employing a staged curriculum (text-to-FOA pretraining, single-operation editing, compound editing) followed by Spatial Preference Optimization (SPO). SPO adapts DPO to the flow-matching context by using edit-specific negative targets (e.g., wrong event, no operation), which effectively addresses common failure modes like source copying or incorrect spatial placement.
Results & evaluation
The experimental evaluation is comprehensive. The authors construct a large-scale synthetic dataset (approx. 375K triplets) using PyRoomAcoustics for controllable FOA rendering, covering 10+ single and compound tasks. They compare against strong baselines including ZETA, AudioEditor, SDEdit, and SmartDJ. The metrics are diverse, covering semantic quality (FD, FAD, KL, LSD), instruction alignment (CLAP), and spatial fidelity (GCC, CRW, FSAD). The results show consistent improvements across all metrics, with particularly strong gains in spatial fidelity (GCC/CRW) compared to stereo-based baselines, validating the benefit of native FOA modeling. The ablation studies are thorough, confirming the necessity of pretraining, the SE-MoE module, and the SPO stage. The comparison between one-stage and multi-stage editing effectively demonstrates the advantage of the proposed one-stage approach in avoiding error accumulation.
Reproducibility
The paper provides high reproducibility. Code and a demo page are available. The data construction process is detailed, including the specific room sizes, direction definitions, and the prompt used for instruction generation. The model architecture details, including the routing regularization losses and CFG scale selection, are provided in the appendix. The use of open-source corpora and standard simulation tools (PyRoomAcoustics) further aids reproducibility.
Limitations
The primary limitation is the reliance on synthetic data generated via room simulation. While controllable, this may not capture the acoustic complexity of real-world recordings (e.g., irregular room responses, microphone artifacts, background noise). The scenes are also limited in duration (~10 seconds) and complexity (2-4 events), which may not generalize to long-form or dense multi-speaker scenes. The human evaluation was conducted with a small group of volunteer graduate students, which may introduce bias, though the objective metrics strongly support the findings.
Broader impact
This work has significant potential for immersive media production, VR/AR, and interactive storytelling. By enabling natural language control over 3D spatial audio, it lowers the barrier to creating complex spatial soundscapes. The one-stage approach is efficient and practical for real-time or near-real-time applications. The framework could be extended to other spatial audio formats (e.g., binaural, higher-order Ambisonics) and integrated into broader audio-visual generation pipelines. The paper introduces a novel one-stage multi-task framework for 3D spatial audio editing that effectively addresses the challenges of preserving unedited regions while executing complex compound instructions. By combining a dual-level Mixture-of-Experts architecture with preference optimization tailored for spatial edits, the method achieves state-of-the-art performance in both semantic and spatial fidelity, marking a significant step forward in controllable spatial audio generation.
Author of this paper? Send a correction.