AudioGS: Spectrogram-Based Audio Gaussian Splatting for Sound Field Reconstruction
Abstract
Spatial audio is fundamental to immersive virtual experiences, yet synthesizing high-fidelity binaural audio from sparse observations remains a significant challenge. Existing methods typically rely on implicit neural representations conditioned on visual priors, which often struggle to capture fine-grained acoustic structures. Inspired by 3D Gaussian Splatting (3DGS), we introduce AudioGS, a novel visual-free framework that explicitly encodes the sound field as a set of Audio Gaussians based on spectrograms. AudioGS associates each time-frequency bin with an Audio Gaussian equipped with dual Spherical Harmonic (SH) coefficients and a decay coefficient. For a target pose, we render binaural audio by evaluating the SH field to capture directionality, incorporating geometry-guided distance attenuation and phase correction, and reconstructing the waveform. Experiments on the Replay-NVAS dataset demonstrate that AudioGS successfully captures complex spatial cues and outperforms state-of-the-art visual-dependent baselines. Specifically, AudioGS reduces the magnitude reconstruction error (MAG) by over 14% and reduces the perceptual quality metric (DPAM) by approximately 25% compared to the best performing visual-guided method.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AudioGS presents a novel approach to binaural audio synthesis that effectively captures spatial cues without relying on visual data. The technical contributions, including the explicit modeling of the sound field and the integration of phase correction, represent a meaningful advancement in the field of audio processing and spatial audio synthesis.
Methodology
The methodology presented in AudioGS is innovative, as it introduces a visual-free framework for synthesizing binaural audio using a set of Audio Gaussians derived from spectrograms. The use of dual Spherical Harmonic coefficients and a decay coefficient to model directional energy and distance attenuation is a significant advancement over existing methods that rely on visual priors. The explicit representation of the sound field allows for more interpretable modeling and captures complex spatial cues effectively. The integration of geometry-guided phase correction further enhances the realism of the synthesized audio, addressing limitations in phase alignment seen in previous methods.
Results & evaluation
The experiments are well-structured, utilizing the Replay-NVAS dataset to evaluate the performance of AudioGS against state-of-the-art visual-dependent methods. The quantitative metrics reported, including MAG and DPAM, provide a robust assessment of audio quality and spatial accuracy. The results demonstrate a significant improvement over existing methods, validating the effectiveness of the proposed approach. The inclusion of subjective listening tests adds depth to the evaluation, confirming the objective metrics with human judgments.
Reproducibility
The paper provides sufficient implementation details, including training setups, loss functions, and evaluation metrics, which facilitates reproducibility. However, the absence of a publicly available code repository or demo URL limits the ease with which other researchers can replicate the results.
Limitations
One limitation of the study is its reliance on a specific dataset (Replay-NVAS), which may not generalize to all acoustic environments. Additionally, while the paper discusses future work on extending the framework to dynamic scenes, the current implementation is limited to static environments, which may restrict its applicability in real-world scenarios.
Broader impact
The potential applications of AudioGS are significant, particularly in immersive technologies such as VR, AR, and XR, where high-fidelity spatial audio is crucial for enhancing user experience. The framework could also benefit fields such as gaming, virtual conferencing, and education, where realistic audio environments are increasingly important. AudioGS presents a novel approach to binaural audio synthesis that effectively captures spatial cues without relying on visual data. The technical contributions, including the explicit modeling of the sound field and the integration of phase correction, represent a meaningful advancement in the field of audio processing and spatial audio synthesis.
Author of this paper? Send a correction.