Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion
Abstract
Spatial audio authoring coordinates source decisions with soundfield synthesis. For editing, plans specify what to change and retain, while reference audio supplies further acoustic detail. We present AMBIT, a unified framework for native First-Order Ambisonics (FOA) generation and editing with three connected contributions. First, a shared Transfusion backbone couples autoregressive planning with rectified-flow rendering through executable ScenePlans that specify source content, activity, and 3D trajectories. For editing, the plan describes the full target scene, including retained sources; contrastive features ground the requested change, while full reference latents supply residual acoustic detail for preservation. Second, an FOA-aware continuous VAE combines shared-decoder omnidirectional recovery, asymmetric grouped KL, and spatial covariance matching to preserve content and inter-channel structure. Third, joint Transfusion on-policy self-distillation (OPSD) refines planning on student-generated prefixes and rendering at sampling states reached under student plans, with target-conditioned teachers supervising both modes of the shared student. Evaluations across speech, music, and sound show improved generation content and spatial quality, acoustic fidelity during instruction-guided editing, and unedited-source preservation over the compared baselines. Project page: https://wjm-george.github.io/AMBIT/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper explicitly states a primary limitation:
Methodology
The paper introduces AMBIT, a unified framework for First-Order Ambisonics (FOA) generation and editing. The core methodology is built upon three interconnected contributions. First, a shared Transfusion backbone effectively couples autoregressive planning with rectified-flow rendering. This is a technically sound approach to integrate discrete planning (ScenePlans) with continuous generation, allowing for explicit control over source content, activity, and 3D trajectories. The ScenePlans are executable and deterministically compiled into semantic and frame-aligned spatial controls, which is a strong design choice for grounding language instructions in 3D audio. For editing, the use of contrastive features for grounding changes and full reference latents for preserving acoustic detail is well-conceived, addressing a key challenge in instruction-guided editing. Second, the proposed FOA-aware continuous Variational Autoencoder (VAE) is a significant improvement over standard audio VAEs for spatial audio. The combination of shared-decoder omnidirectional (W-channel) recovery, asymmetric grouped KL regularization, and especially Spatial Covariance Matching (SCM) directly addresses the unique requirements of FOA data. SCM, by supervising relative energy, coherence, and phase between channels, is a novel and effective way to preserve inter-channel structure crucial for accurate spatial perception. The detailed formulation of SCM, including stabilization and weighting, demonstrates a deep understanding of FOA signal processing. Third, the joint Transfusion On-Policy Self-Distillation (OPSD) mechanism is a sophisticated method for refining both the discrete planner and the continuous renderer. Adapting OPSD to jointly refine both modes, where student plans drive DiT trajectories and target-conditioned teachers supervise on student-generated prefixes and visited sampling states, is a non-trivial technical contribution. This approach addresses the common challenge of distribution shift between training with ground-truth plans and inference with student-generated plans, ensuring that the student learns to perform well in its own operational context. The detailed loss formulations for planning and rendering distillation, including the use of detached weights and target plan memory/audio reference tokens, are well-justified. The overall architecture is complex but appears coherently integrated, leveraging existing strong components (DiT, CLAP) while introducing novel adaptations for spatial audio.
Results & evaluation
The experimental evaluation is comprehensive and rigorous, covering generation, editing, and VAE performance across speech, music, and sound domains. The use of a large dataset (1.6M synthetic and recorded FOA examples, 1M editing pairs, 1M VAE clips) is commendable, although the reliance on synthetic data for a significant portion is a common limitation in spatial audio research. The evaluation metrics are extensive and appropriate for spatial audio, including content-based metrics (paired CLAP, FAD, WER/CER, UTMOS), spatial metrics (directional and binaural errors, GCC, CRW, wCS), and preservation metrics for editing (unchanged-source direction/waveform/level errors). The inclusion of listener ratings in the appendix further strengthens the subjective evaluation. The VAE evaluation clearly demonstrates the benefits of the FOA-aware design, with SCM and grouped KL significantly improving reconstruction quality and spatial accuracy over baselines like DAC, EnCodec, and WavTokenizer. The generation and editing quality results show AMBIT outperforming strong baselines like AudioX-Turbo and SwanWeave across multiple metrics, indicating superior content fidelity, acoustic fidelity, and spatial accuracy. The fixed-renderer comparisons effectively isolate the planner's contribution, while the unedited-source preservation metrics highlight a critical aspect of editing. The post-training OPSD experiments are particularly strong, showing that joint OPSD consistently improves all nine reported metrics over continued supervised training, ReFL, GRPO, and individual branch supervision variants. This provides compelling evidence for the effectiveness of the proposed joint refinement strategy. The detailed ablation studies for the VAE and OPSD components add significant credibility to the individual contributions. The use of Qwen3.5-0.8B as a frozen text backbone and CLAP features for grounding is a practical choice that leverages powerful pre-trained models. The experimental setup, including the use of 8 NVIDIA L40S GPUs, indicates a substantial computational effort.
Reproducibility
The paper provides a good level of detail for reproducibility. The architecture details (15 shared Transfusion blocks, width, attention heads, VAE base width, channel multipliers, strides, latent channels, hop size) are specified. Training details, including learning rates, optimizers (AdamW), batch sizes, loss weights for OPSD, and specific coefficients for various loss terms (STFT, SI-SDR, SCM, phase supervision, etc.), are provided in the main text and appendices. The flow time sampling distributions for training and inference settings (Euler steps, guidance scales) are also given. The datasets are described in detail, including sources (LibriTTS, AudioCaps, AudioSet, FSD50K, MusicCaps, Sphere360, STARSS23, Spatial LibriSpeech) and how synthetic scenes are constructed (Pyroomacoustics, room distributions, source placement, trajectories). The process for generating pseudo-annotations using Qwen3-Omni is also outlined. The project page URL is provided, which typically hosts code, models, and/or audio samples, further aiding reproducibility. While the full code is not directly embedded, the level of detail suggests a strong commitment to enabling replication.
Limitations
The paper explicitly states a primary limitation:
Author of this paper? Send a correction.