CSAVocoder: A Causal Spatial Audio Vocoder Towards Real-Time Spatial Audio Generation
Abstract
Spatial audio vocoders are able to convert mel-spectrograms produced by generative models into spatial audio waveforms. Most neural vocoders are designed for monaural audio, and direct extensions to spatial audio can degrade spatial quality by ignoring inter-channel cues. We present CSAVocoder, a causal GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering. Our framework introduces a Spatial Adaptor that fuses multi-channel mel-spectrograms with dynamic source-listener pose information, together with a spatial consistency discriminator that supervises inter-channel cues. To meet real-time requirements, we design a strictly causal, stateful generator that supports efficient streaming inference with constant memory overhead. Experiments on large-scale spatial audio datasets show that CSAVocoder improves spatial fidelity at competitive audio quality and real-time performance.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[CSAVocoder presents a causal, GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering through a novel Spatial Adaptor and consistency discriminator, achieving real-time streaming performance with improved spatial fidelity.] This paper makes a solid technical contribution to the niche of spatial audio generation by addressing the critical gap of real-time, high-fidelity spatial vocoding. The integration of pose conditioning and inter-channel attention mechanisms provides a robust framework for preserving spatial cues in generated audio, advancing the state-of-the-art in immersive audio synthesis.
Methodology
The paper proposes CSAVocoder, a causal GAN-based spatial audio vocoder. The core methodological contributions include a "Spatial Adaptor" that fuses multi-channel mel-spectrograms with dynamic source-listener pose information (using Fourier features and FiLM conditioning) and a "Spatial Consistency Discriminator" (SCD) that uses axial attention to model inter-channel relationships. The generator is redesigned to be strictly causal with stateful caching for streaming inference. The approach is technically sound, extending HiFi-GAN with specific modules for spatial conditioning and causality. However, the novelty is moderate; the combination of GAN-based vocoding with spatial conditioning is not entirely new (e.g., BinauralFlow, ISDrama), and the specific architectural tweaks (ShuffleUpsample, FiLM) are incremental adaptations of existing techniques to the spatial domain. The use of a dedicated SCD is a notable contribution to the adversarial training process for spatial audio.
Results & evaluation
The experiments are comprehensive, covering binaural and FOA formats. The authors compare against strong baselines including HiFi-GAN, Vocos, DiffWave, and ISDrama. They introduce new spatial consistency metrics (ANG COS, DIS COS) based on Spatial-AST embeddings, which is a valuable addition to the evaluation protocol. The results show improvements in spatial fidelity metrics while maintaining competitive audio quality (PESQ, MRSTFT) and achieving real-time performance (RTF < 1). The ablation studies effectively isolate the contributions of the Mel Adaptor, Position Adaptor, and SCD. The subjective evaluation (MOS-P, MOS-Q) supports the quantitative findings. The comparison with DSP and two-stage baselines clarifies the model's positioning.
Reproducibility
The paper provides detailed architectural configurations, hyperparameters, and loss formulations in the appendix. The dataset sources (MRSSpeech, EasyCom, Spatial LibriSpeech, SoundSpaces) are clearly identified. The preprocessing steps are described. While the code is not linked, the level of detail provided is sufficient for reproduction by researchers in the field.
Limitations
The authors acknowledge limitations in comparing against causal baselines due to implementation differences. They also note that the model is limited to binaural and FOA formats and does not address higher-order ambisonics or personalized HRTFs. The reliance on simulated data for some training sets may introduce domain gaps. The PESQ score is lower than non-causal baselines, which is expected but a trade-off for real-time capability.
Broader impact
This work contributes to the field of immersive audio generation, which has significant applications in VR/AR, gaming, and telepresence. By enabling real-time spatial audio synthesis, it lowers the barrier for interactive applications. The ethical considerations section is thorough, addressing privacy risks associated with pose data and potential misuse for deepfakes. [CSAVocoder presents a causal, GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering through a novel Spatial Adaptor and consistency discriminator, achieving real-time streaming performance with improved spatial fidelity.] This paper makes a solid technical contribution to the niche of spatial audio generation by addressing the critical gap of real-time, high-fidelity spatial vocoding. The integration of pose conditioning and inter-channel attention mechanisms provides a robust framework for preserving spatial cues in generated audio, advancing the state-of-the-art in immersive audio synthesis.
Author of this paper? Send a correction.